2024-09-11T05:37:05.108 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T05:37:05.127 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T05:37:05.211 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581 branch: squid description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writethrough} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900581' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: rbd cache: true rbd cache max dirty: 0 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: daaf78de334647fc0c800f4ac495cc9475a26bae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6175 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 88/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: daaf78de334647fc0c800f4ac495cc9475a26bae targets: smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF90aMa2ytf6ePxlH2Mz5NhmeSmRRSe2Jfn9z0p/8ZAOkuTpt5OuXnopaIwu/Nse7InUD00ilASX6Ei4RpTxN+4= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRFAC+5s3lMu9PqpZqHIIAU/CUL8is2K7E3ck6TOMdVdhLevsM3AAlyOpdkOXuaaGhmoSP39Ert9A5nkG6VXFw= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T05:37:05.211 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa; will attempt to use it 2024-09-11T05:37:05.212 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa/tasks 2024-09-11T05:37:05.212 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T05:37:05.213 INFO:teuthology.task.internal:Checking packages... 2024-09-11T05:37:05.234 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-11T05:37:05.234 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T05:37:05.234 INFO:teuthology.packaging:ref: None 2024-09-11T05:37:05.234 INFO:teuthology.packaging:tag: None 2024-09-11T05:37:05.234 INFO:teuthology.packaging:branch: squid 2024-09-11T05:37:05.234 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:37:05.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-11T05:37:05.447 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:37:05.449 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T05:37:05.456 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T05:37:05.456 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T05:37:05.472 INFO:teuthology.task.internal:Saving configuration 2024-09-11T05:37:05.486 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T05:37:05.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T05:37:05.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:31:15.765588', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIMuGLqmpRpb+jVWX7UyjhpG2vJgiVpOhRHK8B/+civ6'} 2024-09-11T05:37:05.540 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:31:15.764219', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPEW2yf6WubSpNLVKrHxN8VTOUe96PP2Tk01yUOHkGDc'} 2024-09-11T05:37:05.540 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T05:37:05.547 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-11T05:37:05.547 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-11T05:37:05.548 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T05:37:05.605 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4fdd26b370>, signals=[15]) 2024-09-11T05:37:05.605 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T05:37:05.611 INFO:teuthology.task.internal:Opening connections... 2024-09-11T05:37:05.612 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:37:05.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:37:05.696 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:37:05.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:37:05.775 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T05:37:05.781 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-11T05:37:05.787 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-11T05:37:05.787 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="Ubuntu" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="22.04" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_CODENAME=jammy 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:ID=ubuntu 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE=debian 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T05:37:05.839 INFO:teuthology.orchestra.run.smithi165.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T05:37:05.840 INFO:teuthology.orchestra.run.smithi165.stdout:UBUNTU_CODENAME=jammy 2024-09-11T05:37:05.840 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-11T05:37:05.863 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-09-11T05:37:05.868 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-09-11T05:37:05.868 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-09-11T05:37:05.913 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="22.04" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=jammy 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T05:37:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T05:37:05.915 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=jammy 2024-09-11T05:37:05.915 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-09-11T05:37:05.941 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T05:37:05.950 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T05:37:05.955 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T05:37:05.956 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-11T05:37:05.958 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-09-11T05:37:05.963 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T05:37:05.969 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T05:37:05.969 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-11T05:37:06.004 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-09-11T05:37:06.014 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T05:37:06.068 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T05:37:06.081 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T05:37:06.081 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T05:37:06.082 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T05:37:06.082 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T05:37:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:37:06.087 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-11T05:37:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:37:06.088 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-11T05:37:06.138 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-56-generic 2024-09-11T05:37:06.138 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.15.0-56-generic 2024-09-11T05:37:06.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-09-11T05:37:06.139 INFO:teuthology.orchestra.run.smithi165.stdout:5.15.0-56-generic 2024-09-11T05:37:06.140 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.15.0-56-generic 2024-09-11T05:37:06.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-11T05:37:06.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-11T05:37:06.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-11T05:37:06.393 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:37:06.395 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T05:37:06.398 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T05:37:06.401 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:37:06.425 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T05:37:06.475 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T05:37:06.485 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T05:37:06.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T05:37:06.610 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T05:37:06.629 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T05:37:06.677 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T05:37:06.711 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T05:37:06.714 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T05:37:06.716 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T05:37:06.716 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T05:37:06.727 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T05:37:06.732 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T05:37:06.735 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T05:37:06.736 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T05:37:06.736 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T05:37:06.737 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T05:37:06.737 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T05:37:06.737 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T05:37:06.740 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T05:37:06.740 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T05:37:06.744 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T05:37:06.749 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T05:37:06.751 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T05:37:06.756 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T05:37:06.760 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T05:37:06.760 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T05:37:06.767 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T05:37:06.767 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T05:37:06.768 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T05:37:06.768 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T05:37:06.768 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T05:37:06.773 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T05:37:06.776 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T05:37:06.777 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T05:37:06.778 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T05:37:06.778 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T05:37:06.778 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T05:37:06.779 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T05:37:06.779 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T05:37:06.800 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T05:37:06.800 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T05:37:06.801 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T05:37:06.801 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T05:37:06.802 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T05:37:06.803 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T05:37:06.803 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T05:37:06.804 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T05:37:06.805 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T05:37:06.810 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T05:37:06.810 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T05:37:06.815 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T05:37:06.820 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T05:37:06.821 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T05:37:06.822 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T05:37:06.824 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T05:37:06.824 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T05:37:06.834 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T05:37:06.834 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T05:37:06.834 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T05:37:06.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T05:37:06.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T05:37:06.846 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T05:37:06.847 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T05:37:06.847 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T05:37:06.863 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T05:37:06.864 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T05:37:06.864 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T05:37:06.964 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T05:37:07.247 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T05:37:07.419 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T05:37:07.431 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T05:37:07.432 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T05:37:07.517 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T05:37:07.518 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T05:37:07.526 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T05:37:07.526 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T05:37:07.554 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T05:37:07.578 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T05:37:07.581 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T05:37:07.583 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T05:37:07.584 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T05:37:07.665 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T05:37:07.665 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T05:37:12.995 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 15.8 MB in 2s (7,584 kB/s) 2024-09-11T05:37:13.050 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 15.8 MB in 2s (7,471 kB/s) 2024-09-11T05:37:14.397 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:37:14.409 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:37:14.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:37:14.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:37:14.476 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:37:14.490 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:37:14.685 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-11T05:37:14.686 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-11T05:37:14.701 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-11T05:37:14.701 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-11T05:37:14.860 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:37:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:37:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:37:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:37:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T05:37:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: python2.7-minimal 2024-09-11T05:37:14.862 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:37:14.862 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-11T05:37:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:37:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T05:37:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T05:37:14.864 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-11T05:37:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T05:37:14.865 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-11T05:37:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: thermald 2024-09-11T05:37:14.886 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:37:14.887 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:37:14.887 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:37:14.887 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:37:14.887 INFO:teuthology.orchestra.run.smithi172.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T05:37:14.888 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7-minimal 2024-09-11T05:37:14.888 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:37:14.888 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-09-11T05:37:14.889 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:37:14.889 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T05:37:14.889 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T05:37:14.890 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-09-11T05:37:14.890 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T05:37:14.891 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-09-11T05:37:14.891 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2024-09-11T05:37:14.912 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-11T05:37:14.912 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:37:14.912 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T05:37:14.912 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T05:37:14.913 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2024-09-11T05:37:14.914 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T05:37:14.935 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-09-11T05:37:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:37:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T05:37:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T05:37:14.937 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-09-11T05:37:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T05:37:14.965 INFO:teuthology.orchestra.run.smithi165.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:37:14.966 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 113 MB of archives. 2024-09-11T05:37:14.966 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T05:37:14.966 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T05:37:14.988 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:37:14.988 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 113 MB of archives. 2024-09-11T05:37:14.988 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T05:37:14.988 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T05:37:15.404 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T05:37:15.435 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T05:37:15.503 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T05:37:15.551 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T05:37:16.272 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T05:37:16.272 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T05:37:16.272 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T05:37:16.382 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T05:37:16.382 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T05:37:16.382 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T05:37:16.389 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T05:37:16.411 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T05:37:16.506 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T05:37:16.529 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T05:37:16.796 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 113 MB in 1s (76.1 MB/s) 2024-09-11T05:37:16.996 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T05:37:16.998 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 113 MB in 2s (68.8 MB/s) 2024-09-11T05:37:17.157 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T05:37:18.914 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T05:37:18.918 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:19.096 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:19.343 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T05:37:19.347 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:19.595 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:21.116 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T05:37:21.137 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:21.242 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:21.643 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T05:37:21.653 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:21.685 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:21.687 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T05:37:21.708 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:21.799 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:22.282 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T05:37:22.302 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:22.340 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:27.895 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:28.006 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:28.348 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:28.468 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:28.763 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T05:37:28.785 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T05:37:28.829 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:37:29.114 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:29.239 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:29.564 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:29.684 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:29.989 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T05:37:30.011 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T05:37:30.055 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:37:34.137 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T05:37:34.173 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:34.211 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:35.380 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T05:37:35.417 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:37:35.455 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:37.322 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:37.445 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:37.761 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:37:37.866 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:37.967 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T05:37:38.079 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:37:38.282 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:37:38.623 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:37:38.744 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:38.879 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T05:37:39.008 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:39.360 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:37:39.360 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T05:37:39.360 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:37:39.360 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:39.513 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:40.324 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:37:40.324 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T05:37:40.324 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:37:40.324 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:40.467 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:40.922 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:41.893 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:42.725 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T05:37:42.825 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T05:37:42.935 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:42.997 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T05:37:42.997 INFO:teuthology.orchestra.run.smithi165.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:43.411 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T05:37:43.946 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T05:37:44.071 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:37:44.120 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T05:37:44.120 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:57.490 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T05:37:57.490 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub' 2024-09-11T05:37:57.508 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:37:57.525 INFO:teuthology.orchestra.run.smithi165.stdout:Generating grub configuration file ... 2024-09-11T05:37:57.851 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:37:57.870 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:58.245 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:37:58.250 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:37:58.327 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:37:58.333 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:37:58.487 INFO:teuthology.orchestra.run.smithi165.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:37:58.487 INFO:teuthology.orchestra.run.smithi165.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:37:58.487 INFO:teuthology.orchestra.run.smithi165.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:37:58.526 INFO:teuthology.orchestra.run.smithi165.stdout:done 2024-09-11T05:37:58.571 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T05:37:58.571 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2024-09-11T05:37:58.581 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:37:58.600 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2024-09-11T05:37:58.883 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:37:58.909 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:37:58.980 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 21 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-11T05:37:58.999 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.15.0.119.119 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:37:59.000 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-11T05:37:59.000 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T05:37:59.000 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T05:37:59.000 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T05:37:59.001 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T05:37:59.001 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T05:37:59.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-11T05:37:59.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-11T05:37:59.169 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:37:59.175 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:37:59.248 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:37:59.253 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:37:59.301 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:37:59.309 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:37:59.327 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:37:59.420 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:37:59.423 INFO:teuthology.orchestra.run.smithi172.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:37:59.423 INFO:teuthology.orchestra.run.smithi172.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:37:59.423 INFO:teuthology.orchestra.run.smithi172.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:37:59.467 INFO:teuthology.orchestra.run.smithi172.stdout:done 2024-09-11T05:37:59.936 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-09-11T05:37:59.951 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-09-11T05:37:59.951 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-09-11T05:37:59.951 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.119.119 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:37:59.952 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-09-11T05:37:59.953 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T05:37:59.953 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T05:37:59.953 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T05:37:59.953 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T05:37:59.953 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T05:37:59.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-09-11T05:38:00.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-11T05:38:00.216 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:38:00.216 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:38:00.226 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:38:00.243 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:38:00.658 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:38:00.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:38:00.742 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:38:00.952 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-11T05:38:00.952 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-11T05:38:01.142 INFO:teuthology.orchestra.run.smithi165.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T05:38:01.142 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:38:01.142 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:38:01.142 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:38:01.142 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:38:01.143 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:38:01.143 INFO:teuthology.orchestra.run.smithi165.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:38:01.143 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:38:01.185 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:38:01.187 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 21 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.15.0.119.119 2024-09-11T05:38:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:38:01.204 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:38:01.204 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-11T05:38:01.204 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-11T05:38:01.204 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:38:01.204 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-11T05:38:01.204 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-11T05:38:01.250 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.FiEBucOhv6 2024-09-11T05:38:01.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FiEBucOhv6 2024-09-11T05:38:01.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.FiEBucOhv6 2024-09-11T05:38:01.430 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.FiEBucOhv6 is 10KB 2024-09-11T05:38:01.443 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.FiEBucOhv6 2024-09-11T05:38:01.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T05:38:01.501 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:38:01.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T05:38:01.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T05:38:01.568 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T05:38:01.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo update-grub 2024-09-11T05:38:01.659 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:38:01.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:38:01.740 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:38:01.953 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-11T05:38:01.953 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-11T05:38:02.143 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T05:38:02.143 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:38:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:38:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:38:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:38:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:38:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:38:02.144 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:38:02.187 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:38:02.189 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-09-11T05:38:02.201 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.119.119 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:38:02.202 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-09-11T05:38:02.202 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-09-11T05:38:02.248 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.FENMgjVM59 2024-09-11T05:38:02.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FENMgjVM59 2024-09-11T05:38:02.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.FENMgjVM59 2024-09-11T05:38:02.420 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.FENMgjVM59 is 10KB 2024-09-11T05:38:02.433 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.FENMgjVM59 2024-09-11T05:38:02.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T05:38:02.490 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:38:02.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T05:38:02.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T05:38:02.554 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T05:38:02.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2024-09-11T05:38:02.735 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub' 2024-09-11T05:38:02.737 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:38:02.738 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-11T05:38:03.004 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:38:03.009 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:38:03.362 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:38:03.367 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:38:03.412 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:38:03.417 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:38:03.536 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:38:03.536 INFO:teuthology.orchestra.run.smithi165.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:38:03.537 INFO:teuthology.orchestra.run.smithi165.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:38:03.556 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-11T05:38:03.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-11T05:38:03.770 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2024-09-11T05:38:03.771 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:38:03.777 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-09-11T05:38:03.952 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:38:03.957 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:38:04.175 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:38:04.180 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:38:04.219 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:38:04.224 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:38:04.341 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:38:04.341 INFO:teuthology.orchestra.run.smithi172.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:38:04.342 INFO:teuthology.orchestra.run.smithi172.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:38:04.360 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-09-11T05:38:04.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-09-11T05:38:33.588 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T05:38:33.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:38:33.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:38:34.365 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T05:38:34.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:38:34.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:38:52.093 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-11T05:38:52.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-09-11T05:39:01.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:39:01.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:39:01.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:39:01.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:39:04.157 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-11T05:39:04.929 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-09-11T05:39:16.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:39:16.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:39:16.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:39:16.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:40:16.230 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T05:40:16.933 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T05:40:31.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:40:31.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:40:31.475 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-11T05:40:31.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:40:31.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:40:32.018 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:40:32.018 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-11T05:40:32.018 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-11T05:40:32.066 INFO:teuthology.orchestra.run.smithi165.stdout:5.15.0-119-generic 2024-09-11T05:40:32.067 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T05:40:32.067 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T05:40:32.067 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T05:40:32.245 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-11T05:40:32.824 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:40:32.824 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-119-generic"... 2024-09-11T05:40:32.824 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-11T05:40:32.869 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-119-generic 2024-09-11T05:40:32.869 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T05:40:32.869 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T05:40:32.870 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T05:40:33.068 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T05:40:33.068 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T05:40:33.128 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-11T05:40:33.155 DEBUG:teuthology.parallel:result is None 2024-09-11T05:40:33.871 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T05:40:33.871 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T05:40:34.223 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-09-11T05:40:34.255 DEBUG:teuthology.parallel:result is None 2024-09-11T05:40:34.255 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T05:40:34.266 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T05:40:34.266 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T05:40:34.268 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T05:40:34.274 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T05:40:34.285 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T05:40:34.291 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T05:40:34.292 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T05:40:34.315 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T05:40:34.485 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T05:40:34.491 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T05:40:34.492 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T05:40:34.494 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T05:40:34.516 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:40:34.520 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:40:34.524 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:40:34.528 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:40:34.530 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T05:40:34.536 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T05:40:34.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T05:40:34.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T05:40:34.627 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T05:40:34.636 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T05:40:34.636 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T05:40:34.639 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T05:40:34.644 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T05:40:34.690 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T05:40:34.738 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:40:34.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T05:40:34.834 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T05:40:34.840 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T05:40:34.886 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:40:34.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T05:40:34.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-11T05:40:34.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-09-11T05:40:35.029 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T05:40:35.036 INFO:teuthology.task.internal:Starting timer... 2024-09-11T05:40:35.036 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T05:40:35.052 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T05:40:35.061 DEBUG:teuthology.task.selinux:Excluding smithi165: OS 'ubuntu' does not support SELinux 2024-09-11T05:40:35.061 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2024-09-11T05:40:35.061 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T05:40:35.061 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T05:40:35.061 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T05:40:35.061 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T05:40:35.069 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T05:40:35.080 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T05:40:35.081 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi165.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T05:48:04.087 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi165.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-09-11T05:48:04.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:48:04.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:48:04.176 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-11T05:48:04.248 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-11T05:48:04.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:48:04.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:48:04.329 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-11T05:48:04.409 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-11T05:48:04.409 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T05:48:04.421 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T05:48:04.422 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:48:04.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:48:04.425 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:48:04.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:48:04.451 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:48:04.451 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Command line: ntpd -gq 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: ---------------------------------------------------- 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: corporation. Support and training for ntp-4 are 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: available at https://www.nwtime.org/support 2024-09-11T05:48:04.452 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: ---------------------------------------------------- 2024-09-11T05:48:04.453 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: proto: precision = 0.043 usec (-24) 2024-09-11T05:48:04.453 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: basedate set to 2022-02-04 2024-09-11T05:48:04.453 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:48:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:48:04.454 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:48:04.454 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:48:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:48:04.455 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:48:04.455 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:48:04.455 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:48:04.456 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 3 enp3s0f1 172.21.15.165:123 2024-09-11T05:48:04.456 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 4 lo [::1]:123 2024-09-11T05:48:04.456 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f63%5]:123 2024-09-11T05:48:04.456 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:04 ntpd[18457]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Command line: ntpd -gq 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: ---------------------------------------------------- 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: corporation. Support and training for ntp-4 are 2024-09-11T05:48:04.480 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: available at https://www.nwtime.org/support 2024-09-11T05:48:04.481 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: ---------------------------------------------------- 2024-09-11T05:48:04.481 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: proto: precision = 0.062 usec (-24) 2024-09-11T05:48:04.481 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: basedate set to 2022-02-04 2024-09-11T05:48:04.481 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:48:04.482 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:48:04.482 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:48:04.482 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:48:04.483 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:48:04.483 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:48:04.483 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:48:04.484 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:48:04.484 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2024-09-11T05:48:04.484 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 4 lo [::1]:123 2024-09-11T05:48:04.484 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2024-09-11T05:48:04.484 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:04 ntpd[18457]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:48:05.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:05.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:05.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:05 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:06.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:06.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:06.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:06.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:06.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:06.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:06 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:07.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:07.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:07.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:07.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:07.483 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:07.483 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:07.483 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:07.483 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:07 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:08.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:08.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:08.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:08.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:08.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:08.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:08.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:08.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:08.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:08.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:08.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:08.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:08 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:09.454 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:09.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:09.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:09 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:10.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:10.452 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:10.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:10.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:10.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:10.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:10.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:10 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stdout:11 Sep 05:48:11 ntpd[18457]: ntpd: time slew +0.000233 s 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stdout:ntpd: time slew +0.000233s 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:48:11.454 INFO:teuthology.orchestra.run.smithi165.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:48:11.480 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:48:11.481 INFO:teuthology.orchestra.run.smithi172.stdout:11 Sep 05:48:11 ntpd[18457]: ntpd: time slew -0.001171 s 2024-09-11T05:48:11.482 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.001171s 2024-09-11T05:48:11.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:48:11.482 INFO:teuthology.orchestra.run.smithi172.stderr:11 Sep 05:48:11 ntpd[18457]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:48:11.506 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:48:11.507 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-11T05:48:11.507 INFO:teuthology.orchestra.run.smithi165.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.507 INFO:teuthology.orchestra.run.smithi165.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.507 INFO:teuthology.orchestra.run.smithi165.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.507 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:48:11.533 INFO:teuthology.run_tasks:Running task install... 2024-09-11T05:48:11.542 DEBUG:teuthology.task.install:project ceph 2024-09-11T05:48:11.542 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-11T05:48:11.542 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-11T05:48:11.542 INFO:teuthology.task.install:Using flavor: default 2024-09-11T05:48:11.548 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T05:48:11.548 INFO:teuthology.task.install:extra packages: [] 2024-09-11T05:48:11.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-key list | grep Ceph 2024-09-11T05:48:11.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2024-09-11T05:48:11.607 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:48:11.633 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:48:11.635 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:48:11.635 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:48:11.636 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T05:48:11.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:48:11.659 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:48:11.659 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:48:11.660 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T05:48:11.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:48:11.780 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T05:48:11.780 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:48:11.800 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T05:48:11.801 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:48:11.890 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:48:11.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:48:11.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-11T05:48:11.909 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:48:11.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:48:11.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-11T05:48:12.075 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:48:12.086 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:48:12.091 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:48:12.091 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:48:12.103 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:48:12.103 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:48:12.120 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:48:12.127 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:48:12.146 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:48:12.148 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T05:48:12.167 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T05:48:12.170 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:48:12.189 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:48:12.191 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T05:48:12.196 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:48:12.209 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T05:48:12.217 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T05:48:12.236 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T05:48:12.417 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 29.0 kB in 0s (86.0 kB/s) 2024-09-11T05:48:12.448 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 29.0 kB in 0s (84.4 kB/s) 2024-09-11T05:48:13.386 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:48:13.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:48:13.440 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:48:13.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:48:13.456 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-11T05:48:13.518 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-11T05:48:13.644 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-11T05:48:13.644 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-11T05:48:13.727 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-11T05:48:13.727 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-11T05:48:13.830 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:48:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:48:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:48:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:48:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:48:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:48:13.831 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:48:13.831 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-11T05:48:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:48:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:48:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T05:48:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T05:48:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T05:48:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T05:48:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T05:48:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T05:48:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:48:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet zip 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:48:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:48:13.836 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-11T05:48:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: btrfs-tools 2024-09-11T05:48:13.934 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:48:13.934 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:48:13.934 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:48:13.934 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:48:13.935 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:48:13.935 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:48:13.935 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:48:13.935 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-09-11T05:48:13.936 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:48:13.936 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T05:48:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:48:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:48:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet zip 2024-09-11T05:48:13.939 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-09-11T05:48:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:48:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:48:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-09-11T05:48:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2024-09-11T05:48:13.941 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:48:13.942 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:48:13.943 INFO:teuthology.orchestra.run.smithi165.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T05:48:14.041 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T05:48:14.042 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:48:14.043 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:48:14.044 INFO:teuthology.orchestra.run.smithi172.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T05:48:14.081 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:48:14.081 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 181 MB of archives. 2024-09-11T05:48:14.081 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-11T05:48:14.082 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T05:48:14.127 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:48:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:48:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 181 MB of archives. 2024-09-11T05:48:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-11T05:48:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T05:48:14.232 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:48:14.281 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T05:48:14.366 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T05:48:14.383 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T05:48:14.390 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T05:48:14.391 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T05:48:14.395 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T05:48:14.396 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T05:48:14.403 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T05:48:14.406 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T05:48:14.520 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T05:48:14.552 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T05:48:14.555 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:48:14.559 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T05:48:14.560 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T05:48:14.564 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T05:48:14.566 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T05:48:14.568 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:48:14.573 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T05:48:14.615 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T05:48:14.618 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T05:48:14.627 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T05:48:14.648 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:48:14.658 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:48:14.672 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:48:14.752 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:48:14.873 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T05:48:14.876 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T05:48:14.878 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:48:14.885 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T05:48:14.890 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:48:14.919 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:48:14.928 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:48:14.928 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:48:14.930 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:48:14.948 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:48:14.952 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:48:14.956 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:48:14.961 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:48:14.980 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:48:14.992 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:48:15.021 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:48:15.030 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:48:15.031 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:48:15.032 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:48:15.041 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:48:15.045 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:48:15.049 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:48:15.053 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:48:15.054 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:48:15.054 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:48:15.055 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:48:15.057 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:48:15.058 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:48:15.058 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:48:15.058 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:48:15.062 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:48:15.065 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:48:15.124 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:48:15.141 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:48:15.142 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:48:15.142 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:48:15.144 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:48:15.148 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:48:15.148 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:48:15.149 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:48:15.152 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:48:15.155 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:48:15.157 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:48:15.157 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:48:15.206 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:48:15.206 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:48:15.208 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:48:15.209 INFO:teuthology.orchestra.run.smithi165.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:48:15.226 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:48:15.226 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:48:15.227 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:48:15.228 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:48:15.286 INFO:teuthology.orchestra.run.smithi165.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:48:15.289 INFO:teuthology.orchestra.run.smithi165.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:48:15.291 INFO:teuthology.orchestra.run.smithi165.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:48:15.291 INFO:teuthology.orchestra.run.smithi165.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:48:15.303 INFO:teuthology.orchestra.run.smithi165.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:48:15.307 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:48:15.307 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:48:15.308 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:48:15.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:48:15.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:48:15.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:48:15.368 INFO:teuthology.orchestra.run.smithi165.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:48:15.377 INFO:teuthology.orchestra.run.smithi165.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:48:15.378 INFO:teuthology.orchestra.run.smithi165.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:48:15.388 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:48:15.390 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:48:15.391 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:48:15.392 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:48:15.439 INFO:teuthology.orchestra.run.smithi165.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:48:15.439 INFO:teuthology.orchestra.run.smithi165.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T05:48:15.442 INFO:teuthology.orchestra.run.smithi165.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:48:15.449 INFO:teuthology.orchestra.run.smithi165.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:48:15.452 INFO:teuthology.orchestra.run.smithi165.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:48:15.453 INFO:teuthology.orchestra.run.smithi165.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:48:15.458 INFO:teuthology.orchestra.run.smithi165.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:48:15.460 INFO:teuthology.orchestra.run.smithi165.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:48:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:48:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:48:15.472 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:48:15.483 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:48:15.483 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:48:15.484 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:48:15.518 INFO:teuthology.orchestra.run.smithi165.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T05:48:15.520 INFO:teuthology.orchestra.run.smithi165.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T05:48:15.522 INFO:teuthology.orchestra.run.smithi165.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T05:48:15.531 INFO:teuthology.orchestra.run.smithi165.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:48:15.552 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:48:15.552 INFO:teuthology.orchestra.run.smithi165.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T05:48:15.553 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:48:15.557 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:48:15.559 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:48:15.633 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:48:15.634 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:48:15.639 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:48:15.640 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:48:15.641 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:48:15.646 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:48:15.694 INFO:teuthology.orchestra.run.smithi165.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T05:48:15.700 INFO:teuthology.orchestra.run.smithi165.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:48:15.807 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:48:15.809 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:48:15.811 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:48:15.812 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:48:15.814 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:48:15.880 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:48:15.883 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:48:15.884 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:48:15.912 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:48:15.912 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:48:15.974 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:48:15.975 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:48:15.976 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:48:15.977 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:48:15.977 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:48:15.978 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:48:15.985 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:48:16.056 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:48:16.145 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:48:16.146 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:48:16.146 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T05:48:16.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:48:16.161 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:48:16.164 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:48:16.164 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:48:16.175 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:48:16.177 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:48:16.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:48:16.231 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:48:16.267 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:48:16.268 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:48:16.271 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:48:16.273 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:48:16.284 INFO:teuthology.orchestra.run.smithi165.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T05:48:16.284 INFO:teuthology.orchestra.run.smithi165.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T05:48:16.290 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:48:16.291 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:48:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T05:48:16.310 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T05:48:16.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:48:16.318 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:48:16.319 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:48:16.320 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T05:48:16.324 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T05:48:16.331 INFO:teuthology.orchestra.run.smithi165.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T05:48:16.340 INFO:teuthology.orchestra.run.smithi165.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T05:48:16.342 INFO:teuthology.orchestra.run.smithi165.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T05:48:16.384 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T05:48:16.396 INFO:teuthology.orchestra.run.smithi165.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T05:48:16.506 INFO:teuthology.orchestra.run.smithi165.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:48:16.509 INFO:teuthology.orchestra.run.smithi165.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:48:16.512 INFO:teuthology.orchestra.run.smithi165.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:48:16.512 INFO:teuthology.orchestra.run.smithi165.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:48:16.515 INFO:teuthology.orchestra.run.smithi165.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:48:16.557 INFO:teuthology.orchestra.run.smithi165.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:48:16.561 INFO:teuthology.orchestra.run.smithi165.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:48:16.562 INFO:teuthology.orchestra.run.smithi165.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:48:16.593 INFO:teuthology.orchestra.run.smithi165.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T05:48:16.594 INFO:teuthology.orchestra.run.smithi165.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T05:48:16.613 INFO:teuthology.orchestra.run.smithi165.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:48:16.614 INFO:teuthology.orchestra.run.smithi165.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:48:16.670 INFO:teuthology.orchestra.run.smithi165.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:48:16.671 INFO:teuthology.orchestra.run.smithi165.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:48:16.672 INFO:teuthology.orchestra.run.smithi165.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:48:16.673 INFO:teuthology.orchestra.run.smithi165.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:48:16.674 INFO:teuthology.orchestra.run.smithi165.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:48:16.676 INFO:teuthology.orchestra.run.smithi165.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:48:16.683 INFO:teuthology.orchestra.run.smithi165.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:48:16.714 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T05:48:16.751 INFO:teuthology.orchestra.run.smithi165.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:48:16.755 INFO:teuthology.orchestra.run.smithi165.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:48:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:48:16.833 INFO:teuthology.orchestra.run.smithi165.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:48:16.839 INFO:teuthology.orchestra.run.smithi165.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:48:16.842 INFO:teuthology.orchestra.run.smithi165.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:48:16.843 INFO:teuthology.orchestra.run.smithi165.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:48:16.855 INFO:teuthology.orchestra.run.smithi165.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:48:16.857 INFO:teuthology.orchestra.run.smithi165.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:48:16.865 INFO:teuthology.orchestra.run.smithi165.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:48:16.914 INFO:teuthology.orchestra.run.smithi165.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:48:16.916 INFO:teuthology.orchestra.run.smithi165.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:48:16.917 INFO:teuthology.orchestra.run.smithi165.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:48:16.996 INFO:teuthology.orchestra.run.smithi165.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:48:16.998 INFO:teuthology.orchestra.run.smithi165.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:48:17.016 INFO:teuthology.orchestra.run.smithi165.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:48:17.017 INFO:teuthology.orchestra.run.smithi165.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:48:17.018 INFO:teuthology.orchestra.run.smithi165.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:48:17.024 INFO:teuthology.orchestra.run.smithi165.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:48:17.026 INFO:teuthology.orchestra.run.smithi165.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:48:17.755 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T05:48:17.756 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T05:48:17.774 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T05:48:17.816 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T05:48:17.826 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T05:48:17.828 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T05:48:17.882 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T05:48:18.138 INFO:teuthology.orchestra.run.smithi172.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T05:48:18.138 INFO:teuthology.orchestra.run.smithi172.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T05:48:18.277 INFO:teuthology.orchestra.run.smithi165.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T05:48:18.279 INFO:teuthology.orchestra.run.smithi165.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T05:48:18.280 INFO:teuthology.orchestra.run.smithi165.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T05:48:18.711 INFO:teuthology.orchestra.run.smithi165.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T05:48:19.271 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 181 MB in 5s (38.0 MB/s) 2024-09-11T05:48:19.427 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:48:19.434 INFO:teuthology.orchestra.run.smithi172.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T05:48:19.436 INFO:teuthology.orchestra.run.smithi172.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T05:48:19.440 INFO:teuthology.orchestra.run.smithi172.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T05:48:19.518 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:48:19.524 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:48:19.592 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:48:19.628 INFO:teuthology.orchestra.run.smithi172.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T05:48:19.953 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:48:19.979 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:48:20.011 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:48:20.199 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 181 MB in 6s (32.4 MB/s) 2024-09-11T05:48:20.331 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:48:20.357 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:48:20.357 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:48:20.397 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:48:20.457 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:48:20.463 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:48:20.548 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:48:20.826 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:48:20.852 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:20.873 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:48:20.900 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:48:20.900 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:20.934 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:48:21.246 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:48:21.261 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:48:21.272 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:48:21.288 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:21.312 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:48:21.320 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:21.623 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:48:21.649 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:21.689 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:21.716 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:48:21.742 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:21.784 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:22.017 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:48:22.032 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:48:22.067 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:48:22.163 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:48:22.189 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:22.229 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:22.361 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librados2. 2024-09-11T05:48:22.388 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:22.428 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:22.566 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:48:22.592 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:48:22.632 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:48:22.864 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:22.930 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:22.969 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:48:22.995 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:48:23.027 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:48:23.297 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librados2. 2024-09-11T05:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:23.383 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:48:23.409 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:23.449 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:23.702 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:48:23.717 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:23.752 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:23.791 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:48:23.817 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:23.857 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.038 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:48:24.064 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.277 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:48:24.303 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:24.343 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.373 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:48:24.400 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:24.440 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.596 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:48:24.611 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:24.646 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.734 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:48:24.760 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:24.801 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:24.967 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:48:24.993 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:25.032 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:25.128 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:48:25.155 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:48:25.195 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:48:25.284 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:48:25.299 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:25.351 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:25.506 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:48:25.532 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:48:25.564 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:48:25.662 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:48:25.688 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:25.729 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:25.800 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:48:25.815 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:25.858 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:26.023 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:48:26.050 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:48:26.082 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:48:26.203 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:48:26.228 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:48:26.269 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:48:26.368 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:48:26.394 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:48:26.546 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:48:26.572 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:48:26.613 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:48:26.696 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:48:26.722 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:26.762 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:26.899 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:48:26.915 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:48:26.949 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:48:27.107 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:48:27.133 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:48:27.173 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:48:27.260 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:48:27.286 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:48:27.327 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:48:27.434 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:48:27.449 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:48:27.492 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:48:27.846 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:48:27.862 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:48:27.896 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:48:27.954 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:48:27.980 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:48:28.020 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:48:28.132 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package zip. 2024-09-11T05:48:28.148 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:48:28.190 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:48:28.349 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:48:28.375 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:48:28.415 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:48:28.502 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package unzip. 2024-09-11T05:48:28.528 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:48:28.568 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:48:28.685 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:48:28.700 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:48:28.743 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:48:28.996 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package zip. 2024-09-11T05:48:29.011 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:48:29.013 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:48:29.040 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:48:29.079 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:48:29.080 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:48:29.391 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package unzip. 2024-09-11T05:48:29.412 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:48:29.417 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:48:29.427 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:29.457 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:48:29.462 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:29.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:48:29.837 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:48:29.869 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:48:30.023 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:48:30.038 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:30.081 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:30.227 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:48:30.254 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:30.294 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:30.417 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:48:30.444 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:48:30.484 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:48:30.769 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:48:30.785 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:30.819 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:30.829 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:48:30.845 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:30.888 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:31.156 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:48:31.182 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:31.222 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:31.224 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:48:31.250 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:48:31.291 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:48:31.585 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:48:31.612 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:31.652 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:32.072 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:48:32.098 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:32.611 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:48:32.637 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:32.695 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:33.273 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:48:33.299 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:48:33.339 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:48:33.617 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:48:33.644 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:48:33.661 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:48:33.683 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:48:33.687 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:33.728 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:33.945 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:48:33.971 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:48:34.003 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:48:34.305 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:48:34.332 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:48:34.372 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:48:34.398 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:48:34.424 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:48:34.465 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:48:34.683 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:48:34.710 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:48:34.750 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:48:34.767 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:48:34.794 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:48:34.834 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:48:35.044 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:48:35.071 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:48:35.111 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:48:35.162 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:48:35.189 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:48:35.229 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:48:35.354 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:48:35.370 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:48:35.404 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:48:35.465 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:48:35.480 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:48:35.523 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:48:35.716 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:48:35.743 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:48:35.782 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:48:35.826 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:48:35.852 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:48:35.892 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:48:36.085 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:48:36.111 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:48:36.151 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:48:36.196 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:48:36.222 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:48:36.262 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:48:36.488 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:48:36.514 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:48:36.523 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:48:36.550 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:48:36.554 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:48:36.582 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:48:36.866 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:48:36.868 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:48:36.892 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:48:36.894 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:48:36.932 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:48:36.934 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:48:37.235 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:48:37.256 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:48:37.261 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:48:37.282 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:48:37.301 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:48:37.329 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:48:37.562 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:48:37.577 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:48:37.612 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:48:37.658 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:48:37.684 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:48:37.724 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:48:37.907 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:48:37.934 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:48:37.974 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:48:38.261 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:48:38.285 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:48:38.287 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:48:38.312 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:48:38.327 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:48:38.351 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:48:38.630 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:48:38.656 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:48:38.670 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:48:38.697 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:48:38.697 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:48:38.737 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:48:38.958 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:48:38.984 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:48:39.016 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:48:39.073 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:48:39.100 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:48:39.143 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:48:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:48:39.338 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:48:39.378 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:48:39.426 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:48:39.441 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:48:39.475 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:48:39.681 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:48:39.707 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:48:39.728 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:48:39.747 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:48:39.755 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:48:39.795 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:48:40.034 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:48:40.062 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:48:40.092 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:48:40.106 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:48:40.129 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:48:40.161 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:48:40.387 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:48:40.425 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:48:40.428 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:48:40.450 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:48:40.453 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:48:40.495 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:48:40.739 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:48:40.765 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:48:40.794 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:48:40.805 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:48:40.821 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:48:40.861 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:48:41.109 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:48:41.135 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:48:41.172 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:48:41.175 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:48:41.199 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:48:41.238 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:48:41.537 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:48:41.563 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:48:41.593 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:48:41.603 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:48:41.620 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:48:41.660 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:48:41.881 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:48:41.896 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:48:41.931 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:48:41.945 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:48:41.972 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:48:42.012 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:48:42.201 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:48:42.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:48:42.265 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:48:42.267 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:48:42.291 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:48:42.348 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:48:42.604 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:48:42.618 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:48:42.631 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:48:42.644 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:48:42.670 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:48:42.684 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:48:43.012 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:48:43.025 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:48:43.038 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:48:43.052 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:48:43.092 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:48:43.123 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:48:43.437 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:48:43.463 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:48:43.503 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:48:43.531 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:48:43.546 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:43.589 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:43.789 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:48:43.804 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:48:43.839 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:48:43.913 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:48:43.940 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:43.971 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:44.117 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:48:44.132 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:48:44.167 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:48:44.273 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:48:44.300 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:48:44.340 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:48:44.470 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:48:44.497 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:48:44.537 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:48:44.651 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:48:44.678 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:48:44.717 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:48:44.798 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:48:44.825 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:44.856 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:45.045 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:48:45.072 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:45.112 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:45.245 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:48:45.271 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:45.312 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:45.456 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:48:45.483 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:45.523 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:45.597 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:48:45.624 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:48:45.655 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:48:45.934 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:48:45.960 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:48:46.000 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:48:46.118 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:48:46.144 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:46.184 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:46.337 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:48:46.363 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:46.403 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:46.781 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:48:46.807 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:46.847 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:47.212 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph. 2024-09-11T05:48:47.228 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:47.270 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:47.401 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:48:47.427 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:47.467 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:48.307 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:48:48.334 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:48.365 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:48.726 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:48:48.753 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:48.793 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:49.048 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2024-09-11T05:48:49.061 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:49.104 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:49.196 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:48:49.222 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:49.262 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:49.440 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:48:49.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:49.506 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:49.514 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:48:49.541 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:48:49.581 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:48:49.867 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:48:49.876 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:48:49.893 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:48:49.903 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:49.933 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:48:49.943 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:50.262 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:48:50.278 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:48:50.321 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:50.624 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:48:50.651 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:48:50.690 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:48:50.959 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:48:50.963 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:48:50.975 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:48:50.990 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:48:51.017 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:48:51.029 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:48:51.332 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:48:51.347 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:48:51.382 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:48:51.752 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:48:51.778 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:51.818 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:52.056 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:48:52.084 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:48:52.123 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:48:52.155 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:48:52.181 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:48:52.221 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:48:52.451 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:48:52.477 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:48:52.517 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:48:52.523 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:48:52.550 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:48:52.590 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:48:52.870 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:48:52.897 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:52.918 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:48:52.937 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:52.945 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:52.984 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:53.273 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:48:53.300 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:48:53.340 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:48:53.693 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:48:53.719 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:48:53.759 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:48:54.104 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:48:54.131 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:54.171 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:54.231 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:48:54.258 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:48:54.297 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:48:54.784 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:48:54.811 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:48:54.850 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:48:55.229 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:48:55.256 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:48:55.295 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:48:55.434 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:48:55.461 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:48:55.500 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:48:55.598 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:48:55.625 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:48:55.664 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:48:55.989 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:48:56.015 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:48:56.060 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:48:56.314 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:48:56.341 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:56.374 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:48:56.392 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:48:56.424 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:48:56.457 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:56.710 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:48:56.737 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:48:56.776 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:48:57.311 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:48:57.338 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:48:57.377 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:48:57.502 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:48:57.529 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:48:57.646 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:48:57.680 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:48:57.707 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:48:57.746 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:48:58.066 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:48:58.093 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:48:58.132 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:48:58.411 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:48:58.438 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:48:58.583 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:48:58.610 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:48:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:48:58.652 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:48:58.980 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:48:59.008 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:48:59.047 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:48:59.169 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:48:59.196 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:48:59.235 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:48:59.350 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:48:59.377 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:48:59.416 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:48:59.538 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:48:59.562 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:48:59.597 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:48:59.736 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:48:59.763 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:48:59.802 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:48:59.908 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:48:59.935 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:48:59.974 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:49:00.097 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:49:00.124 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:49:00.163 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:49:00.264 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:49:00.291 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:49:00.330 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:49:00.650 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:49:00.677 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:49:00.716 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:49:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:49:00.855 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:49:00.893 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:01.028 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:49:01.055 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:49:01.094 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:49:01.213 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:49:01.240 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:49:01.279 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:49:01.414 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:49:01.442 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:49:01.481 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:49:01.565 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:49:01.592 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:49:01.632 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:49:01.859 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package jq. 2024-09-11T05:49:01.875 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:49:01.917 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:49:02.170 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package socat. 2024-09-11T05:49:02.178 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:49:02.194 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:49:02.197 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:49:02.236 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:02.236 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:49:02.531 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:49:02.547 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:49:02.556 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:49:02.583 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:49:02.589 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:49:02.622 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:49:02.883 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:49:02.911 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:49:02.941 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:49:02.943 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:49:02.971 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:03.010 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:03.202 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2024-09-11T05:49:03.230 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:49:03.269 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:49:03.521 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2024-09-11T05:49:03.537 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:49:03.588 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:49:03.891 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:49:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:49:03.949 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:49:04.296 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:49:04.323 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:04.362 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:05.289 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:49:05.316 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:49:05.355 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:05.659 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:49:05.686 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:05.726 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:05.995 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:49:06.011 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:49:06.045 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:49:06.349 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:49:06.376 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:49:06.415 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:49:06.533 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:49:06.549 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:49:06.583 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:06.786 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:49:06.813 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:49:06.852 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:49:06.870 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:49:06.898 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:06.928 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:07.240 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:49:07.267 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:49:07.282 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:49:07.306 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:49:07.309 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:49:07.348 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:49:07.610 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:49:07.637 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:49:07.644 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:49:07.672 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:49:07.676 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:49:07.710 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:49:08.065 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:49:08.092 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:49:08.096 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:49:08.123 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:49:08.131 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:49:08.162 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:49:08.465 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:49:08.492 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:49:08.531 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:49:08.544 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:49:08.572 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:49:08.611 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:49:08.836 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:49:08.850 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:49:08.872 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:49:08.884 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:49:08.888 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:49:08.930 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:49:09.537 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:49:09.550 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:49:09.587 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:49:09.742 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:49:09.769 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:49:09.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:49:09.873 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:49:09.900 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:49:09.939 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:49:10.127 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:49:10.155 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:49:10.194 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:49:10.259 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:49:10.287 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:49:10.317 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:49:10.497 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:49:10.525 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:49:10.555 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:49:10.738 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:49:10.754 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:49:10.787 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:49:10.842 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:49:10.870 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:49:10.908 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:49:11.082 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:49:11.109 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:49:11.148 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:49:11.203 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:49:11.231 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:49:11.269 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:49:11.459 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:49:11.488 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:49:11.526 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:49:11.548 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:49:11.576 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:49:11.614 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:49:11.846 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:49:11.874 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:49:11.912 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:49:12.060 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:49:12.088 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:49:12.126 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:49:12.181 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:49:12.198 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:49:12.239 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:49:12.437 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:49:12.465 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:49:12.503 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:49:12.782 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:49:12.810 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:49:12.840 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:49:12.943 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:49:12.971 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:13.010 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:13.185 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:49:13.213 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:49:13.251 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:49:13.520 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:49:13.549 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:49:13.587 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:49:13.787 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:49:13.803 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:13.837 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:14.108 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:49:14.316 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:49:14.344 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:14.382 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:14.417 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:49:14.551 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:49:14.812 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:49:15.121 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:49:15.194 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:49:15.210 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:49:15.252 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:15.537 INFO:teuthology.orchestra.run.smithi165.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:49:15.545 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:49:15.603 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:15.799 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:49:15.950 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:49:16.094 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user cephadm....done 2024-09-11T05:49:16.209 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:49:16.225 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:49:16.502 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:49:16.575 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:49:16.753 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:49:16.879 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:49:16.959 INFO:teuthology.orchestra.run.smithi172.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:49:17.027 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:17.154 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:49:17.414 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:49:17.531 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:49:17.551 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2024-09-11T05:49:17.674 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:49:17.816 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:49:17.926 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:49:18.177 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:49:18.193 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:49:18.328 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:49:18.453 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:49:18.571 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:49:18.737 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:49:18.831 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:49:18.948 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:49:18.980 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:49:19.225 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:49:19.231 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:49:19.466 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:49:19.585 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:49:19.700 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:19.836 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:49:19.951 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:49:20.077 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:49:20.104 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:49:20.207 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:49:20.337 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:49:20.372 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:49:20.613 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:49:20.631 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:49:20.891 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:49:21.007 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:49:21.036 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:49:21.108 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:49:21.133 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:21.234 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:49:21.393 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:49:21.494 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:49:21.593 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:49:21.603 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:49:21.719 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:49:21.720 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:49:21.987 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:49:21.988 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:49:22.279 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:49:22.379 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:49:22.428 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:49:22.492 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:49:22.573 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:49:22.618 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:49:22.682 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:49:22.969 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:49:23.095 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:49:23.317 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:49:23.363 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:49:23.559 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:49:23.639 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:49:23.685 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:49:23.811 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:49:23.948 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:49:24.066 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:49:24.112 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:49:24.497 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:49:24.768 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:49:24.898 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:49:25.186 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:49:25.286 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:49:25.299 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:49:25.412 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:49:25.584 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:49:25.730 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:49:25.944 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:49:26.106 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:49:26.195 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:49:26.320 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:26.349 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:49:26.638 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:49:26.751 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:49:26.931 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:49:27.035 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:49:27.362 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:49:27.613 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:49:27.739 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:28.023 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:49:28.139 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:49:28.341 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:49:28.365 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:28.500 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:49:28.759 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:49:28.894 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:49:29.020 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:49:29.305 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:49:29.472 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:49:29.539 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:49:29.690 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:29.765 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:29.799 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:49:29.891 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:49:30.067 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:49:30.176 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:49:30.209 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:49:30.294 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:49:30.470 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:49:30.579 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:49:30.745 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:49:30.838 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:49:30.997 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:49:31.081 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:31.182 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:49:31.231 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:49:31.466 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:49:31.490 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:49:31.616 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:49:31.717 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:49:31.893 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:49:31.960 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:49:32.027 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:49:32.211 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:49:32.328 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:49:32.471 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:49:32.630 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:49:32.747 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:49:32.873 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:49:32.873 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:49:32.973 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:49:33.074 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:49:33.149 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:49:33.156 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:49:33.275 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:49:33.385 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:49:33.519 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:49:33.568 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:49:33.844 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:49:33.904 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:49:34.078 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:49:34.155 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:49:34.204 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:49:34.256 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:49:34.314 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:49:34.386 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:49:34.516 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:34.642 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:49:34.657 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:49:34.783 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:49:34.968 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:49:35.168 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:49:35.303 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:49:35.427 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:49:35.545 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:49:35.579 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:49:35.688 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:49:35.804 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:49:35.914 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:49:35.956 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:49:36.215 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:49:36.231 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:49:36.524 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:49:36.552 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:49:36.809 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:49:36.929 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:49:36.935 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:49:37.202 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:49:37.205 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:49:37.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:49:37.516 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:49:37.642 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:37.777 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:37.815 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:49:37.903 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:49:38.200 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:49:38.222 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:38.323 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:38.424 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:49:38.476 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:49:38.693 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:38.796 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:49:38.827 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:38.914 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:38.953 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.040 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.141 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:49:39.166 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:49:39.166 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:49:39.468 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.594 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.638 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.720 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:49:39.739 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.848 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:39.974 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:49:40.005 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:40.131 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:40.449 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:40.663 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:49:40.663 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:49:41.151 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.228 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.251 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.354 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.455 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.478 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:49:41.556 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:41.658 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:42.201 INFO:teuthology.orchestra.run.smithi165.stdout:Adding group ceph....done 2024-09-11T05:49:42.536 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user ceph....done 2024-09-11T05:49:42.704 INFO:teuthology.orchestra.run.smithi165.stdout:Setting system user ceph properties....done 2024-09-11T05:49:42.717 INFO:teuthology.orchestra.run.smithi165.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:49:42.724 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:42.855 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:49:42.875 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:42.984 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:43.102 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:43.193 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:49:43.203 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:43.659 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:43.729 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2024-09-11T05:49:43.835 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:44.072 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2024-09-11T05:49:44.241 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2024-09-11T05:49:44.260 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:49:44.310 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:49:44.310 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:49:44.404 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:49:44.724 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:49:44.754 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:45.071 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:49:45.204 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:45.315 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:45.531 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:45.773 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:49:45.774 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:49:45.861 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:49:45.861 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:49:46.250 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:46.333 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:46.551 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:49:46.551 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:49:46.592 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:49:47.009 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:47.043 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:47.276 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:49:47.276 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:49:47.287 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:49:47.287 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:49:47.737 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:47.761 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:47.887 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:47.946 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:49:47.946 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:49:48.021 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:48.252 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:49:48.253 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:49:48.396 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:48.639 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:49:48.640 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:49:48.723 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:48.866 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:48.992 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:49.089 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:49.143 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:49.215 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:49.358 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:49.496 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:49:49.571 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:49:49.571 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:49:49.605 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:49:49.741 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:49:50.051 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:50.185 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:50.311 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:50.462 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:49:50.816 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:49:50.868 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:49:50.916 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:49:51.077 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:49:51.510 INFO:teuthology.orchestra.run.smithi165.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:49:51.516 DEBUG:teuthology.parallel:result is None 2024-09-11T05:49:52.152 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:49:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:49:52.791 DEBUG:teuthology.parallel:result is None 2024-09-11T05:49:52.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:49:52.930 DEBUG:teuthology.orchestra.run.smithi165:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:49:52.949 INFO:teuthology.orchestra.run.smithi165.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:49:52.950 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:49:52.950 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T05:49:52.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:49:53.088 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:49:53.107 INFO:teuthology.orchestra.run.smithi172.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:49:53.107 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:49:53.107 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T05:49:53.109 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T05:49:53.109 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:53.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:49:53.123 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:53.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:49:53.162 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T05:49:53.162 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:53.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:49:53.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:49:53.235 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:53.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:49:53.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:49:53.301 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T05:49:53.301 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:53.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:49:53.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:49:53.367 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:53.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:49:53.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:49:53.433 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T05:49:53.433 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:53.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:49:53.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:49:53.499 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:53.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:49:53.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:49:53.564 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T05:49:53.640 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T05:49:53.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-09-11T05:49:53.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-09-11T05:49:53.654 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T05:49:53.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:49:53.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:49:53.709 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T05:49:53.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:49:53.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:49:53.774 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T05:49:53.774 INFO:tasks.ceph:config {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T05:49:53.775 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581', 'branch': 'squid', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writethrough}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900581', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi165.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF90aMa2ytf6ePxlH2Mz5NhmeSmRRSe2Jfn9z0p/8ZAOkuTpt5OuXnopaIwu/Nse7InUD00ilASX6Ei4RpTxN+4=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBRFAC+5s3lMu9PqpZqHIIAU/CUL8is2K7E3ck6TOMdVdhLevsM3AAlyOpdkOXuaaGhmoSP39Ert9A5nkG6VXFw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T05:49:53.775 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T05:49:53.805 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T05:49:53.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:49:53.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:49:53.869 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:53.869 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:49:53.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:49:53.912 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-11 05:44:30.898873003 +0000 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-11 05:44:30.646875254 +0000 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-11 05:44:30.646875254 +0000 2024-09-11T05:49:53.960 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-11T05:49:53.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:49:54.014 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-11T05:49:54.014 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-11T05:49:54.014 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000412163 s, 1.2 MB/s 2024-09-11T05:49:54.015 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:49:54.066 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-11 05:44:31.390868609 +0000 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-11 05:44:31.154870717 +0000 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-11 05:44:31.154870717 +0000 2024-09-11T05:49:54.116 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-11T05:49:54.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:49:54.174 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-11T05:49:54.174 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-11T05:49:54.174 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000414034 s, 1.2 MB/s 2024-09-11T05:49:54.175 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:49:54.229 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-11 05:44:31.874864286 +0000 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-11 05:44:31.618866572 +0000 2024-09-11T05:49:54.276 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-11 05:44:31.618866572 +0000 2024-09-11T05:49:54.277 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-11T05:49:54.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:49:54.330 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-11T05:49:54.330 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-11T05:49:54.330 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000363054 s, 1.4 MB/s 2024-09-11T05:49:54.331 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:49:54.382 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-11 05:44:32.362859926 +0000 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-11 05:44:32.110862178 +0000 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-11 05:44:32.110862178 +0000 2024-09-11T05:49:54.428 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-11T05:49:54.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:49:54.482 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-11T05:49:54.482 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-11T05:49:54.482 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000386673 s, 1.3 MB/s 2024-09-11T05:49:54.483 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:49:54.533 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:54.533 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:54.533 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:49:54.539 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:49:54.539 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-09-11T05:49:54.585 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-11 05:44:34.558388369 +0000 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-11 05:44:34.314390549 +0000 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-11 05:44:34.314390549 +0000 2024-09-11T05:49:54.586 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-11T05:49:54.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:49:54.640 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-11T05:49:54.640 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-11T05:49:54.640 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000409489 s, 1.3 MB/s 2024-09-11T05:49:54.641 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:49:54.690 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-09-11T05:49:54.737 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:49:54.737 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-11 05:44:35.054383938 +0000 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-11 05:44:34.782386368 +0000 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-11 05:44:34.782386368 +0000 2024-09-11T05:49:54.738 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-11T05:49:54.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:49:54.790 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-11T05:49:54.791 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-11T05:49:54.791 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000312956 s, 1.6 MB/s 2024-09-11T05:49:54.792 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:49:54.839 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-09-11T05:49:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-11 05:44:35.538379616 +0000 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-11 05:44:35.298381759 +0000 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-11 05:44:35.298381759 +0000 2024-09-11T05:49:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-11T05:49:54.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:49:54.939 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-11T05:49:54.939 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-11T05:49:54.939 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000318002 s, 1.6 MB/s 2024-09-11T05:49:54.940 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:49:54.987 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-09-11T05:49:55.033 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-11 05:44:36.026375257 +0000 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-11 05:44:35.782377436 +0000 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-11 05:44:35.782377436 +0000 2024-09-11T05:49:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-11T05:49:55.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:49:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-11T05:49:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-11T05:49:55.088 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000334749 s, 1.5 MB/s 2024-09-11T05:49:55.089 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:49:55.139 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:55.139 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-11T05:49:55.139 INFO:tasks.ceph:Generating config... 2024-09-11T05:49:55.140 INFO:tasks.ceph:[client] rbd cache = True 2024-09-11T05:49:55.140 INFO:tasks.ceph:[client] rbd cache max dirty = 0 2024-09-11T05:49:55.141 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T05:49:55.141 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T05:49:55.141 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T05:49:55.141 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T05:49:55.141 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T05:49:55.141 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T05:49:55.141 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T05:49:55.141 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T05:49:55.141 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-11T05:49:55.141 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-11T05:49:55.141 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-11T05:49:55.141 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T05:49:55.141 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T05:49:55.142 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T05:49:55.143 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T05:49:55.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T05:49:55.175 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T05:49:55.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T05:49:55.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:49:55.323 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.165'), ('mon.c', '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]'), ('mon.b', '172.21.15.172')] 2024-09-11T05:49:55.323 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],172.21.15.172', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd cache': True, 'rbd cache max dirty': 0}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-11T05:49:55.324 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:55.324 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:49:55.368 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.165 --addv c '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]' --add b 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:49:55.454 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: generated fsid d5628057-c743-43f5-a51d-1915a16ccfdd 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:setting min_mon_release = quincy 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:epoch 0 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:fsid d5628057-c743-43f5-a51d-1915a16ccfdd 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:last_changed 2024-09-11T05:49:55.455825+0000 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:created 2024-09-11T05:49:55.455825+0000 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:min_mon_release 17 (quincy) 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:election_strategy: 1 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:0: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.a 2024-09-11T05:49:55.455 INFO:teuthology.orchestra.run.smithi165.stdout:1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-11T05:49:55.456 INFO:teuthology.orchestra.run.smithi165.stdout:2: [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] mon.c 2024-09-11T05:49:55.456 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T05:49:55.457 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:49:55.504 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d5628057-c743-43f5-a51d-1915a16ccfdd... 2024-09-11T05:49:55.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T05:49:55.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T05:49:55.577 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-09-11T05:49:55.577 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.578 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.579 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:49:55.580 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],172.21.15.172" 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon client directed command retry = 5 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on pool no app = False 2024-09-11T05:49:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject socket failures = 5000 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = d5628057-c743-43f5-a51d-1915a16ccfdd 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-09-11T05:49:55.582 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug trim objects = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.583 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: bdev async discard = True 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: bdev enable discard = True 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore allocator = stupid 2024-09-11T05:49:55.584 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 1/20 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 1/20 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 4/10 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: osd blocked scrub grace period = 3600 2024-09-11T05:49:55.585 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: osd max markdown count = 1000 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: osd shutdown pgref assert = True 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-09-11T05:49:55.586 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-09-11T05:49:55.587 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: rbd cache = True 2024-09-11T05:49:55.589 INFO:teuthology.orchestra.run.smithi165.stdout: rbd cache max dirty = 0 2024-09-11T05:49:55.590 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-09-11T05:49:55.590 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-09-11T05:49:55.590 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-09-11T05:49:55.590 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-09-11T05:49:55.590 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.591 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.592 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:49:55.593 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],172.21.15.172" 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on pool no app = False 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2024-09-11T05:49:55.594 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = d5628057-c743-43f5-a51d-1915a16ccfdd 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-09-11T05:49:55.595 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.596 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: bdev async discard = True 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: bdev enable discard = True 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = stupid 2024-09-11T05:49:55.597 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:49:55.598 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-09-11T05:49:55.599 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-09-11T05:49:55.600 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:49:55.601 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: rbd cache = True 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout: rbd cache max dirty = 0 2024-09-11T05:49:55.602 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-09-11T05:49:55.603 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2024-09-11T05:49:55.603 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2024-09-11T05:49:55.604 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T05:49:55.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T05:49:55.693 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T05:49:55.694 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:55.694 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T05:49:55.699 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:55.699 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T05:49:55.748 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:49:55.748 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:55.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T05:49:55.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:49:55.812 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:49:55.812 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:49:55.860 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:49:55.860 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:55.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T05:49:55.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:49:55.884 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:49:55.884 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:49:55.929 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T05:49:55.930 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T05:49:55.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-11T05:49:55.973 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-11T05:49:55.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T05:49:56.020 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T05:49:56.037 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T05:49:56.040 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T05:49:56.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T05:49:56.074 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T05:49:56.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T05:49:56.120 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-11T05:49:56.130 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T05:49:56.130 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-11T05:49:56.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T05:49:56.143 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:56.143 INFO:tasks.ceph:role: osd.0 2024-09-11T05:49:56.143 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:49:56.143 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:49:56.208 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:49:56.660 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-11T05:49:56.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-11T05:49:56.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T05:49:56.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T05:49:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:49:56.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:49:56.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T05:49:56.825 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:56.825 INFO:tasks.ceph:role: osd.1 2024-09-11T05:49:56.825 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:49:56.825 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:49:56.888 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:49:56.889 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:49:56.889 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:49:57.358 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-11T05:49:57.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-11T05:49:57.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T05:49:57.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T05:49:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:49:57.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:49:57.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T05:49:57.567 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:57.567 INFO:tasks.ceph:role: osd.2 2024-09-11T05:49:57.567 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:49:57.567 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:49:57.631 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:49:58.124 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-11T05:49:58.126 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-11T05:49:58.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T05:49:58.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T05:49:58.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:49:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:49:58.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T05:49:58.356 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T05:49:58.356 INFO:tasks.ceph:role: osd.3 2024-09-11T05:49:58.356 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:49:58.356 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:49:58.419 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:49:58.836 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-11T05:49:58.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-11T05:49:58.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-11T05:49:58.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T05:49:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:49:58.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:49:58.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:49:59.047 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:49:59.042+0000 7ff2039567c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T05:49:59.050 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:49:59.046+0000 7ff2039567c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T05:49:59.051 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:49:59.046+0000 7ff2039567c0 -1 bdev(0x560b93fb7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T05:49:59.051 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:49:59.046+0000 7ff2039567c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T05:50:02.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T05:50:02.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:02.309 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:02.306+0000 7f3ae52b57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T05:50:02.310 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:02.306+0000 7f3ae52b57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T05:50:02.310 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:02.306+0000 7f3ae52b57c0 -1 bdev(0x55c9de9c1800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:02.310 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:02.306+0000 7f3ae52b57c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T05:50:05.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T05:50:05.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:05.578 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:05.574+0000 7f1e9fe587c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T05:50:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:05.574+0000 7f1e9fe587c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T05:50:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:05.574+0000 7f1e9fe587c0 -1 bdev(0x5621b917d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:05.574+0000 7f1e9fe587c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T05:50:08.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T05:50:09.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:09.106 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:09.102+0000 7f11a96767c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T05:50:09.107 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:09.102+0000 7f11a96767c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T05:50:09.107 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:09.102+0000 7f11a96767c0 -1 bdev(0x55d1de1eb800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:09.107 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-11T05:50:09.102+0000 7f11a96767c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T05:50:12.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T05:50:12.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T05:50:12.335 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T05:50:12.335 INFO:tasks.ceph:role: osd.4 2024-09-11T05:50:12.335 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:50:12.335 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:50:12.397 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:50:12.398 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:50:12.398 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:50:12.398 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:50:12.398 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:50:12.846 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-09-11T05:50:12.848 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-11T05:50:12.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-11T05:50:12.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T05:50:12.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:50:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:50:12.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T05:50:13.016 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T05:50:13.016 INFO:tasks.ceph:role: osd.5 2024-09-11T05:50:13.017 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:50:13.017 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:50:13.081 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:50:13.081 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:50:13.082 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:50:13.546 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-09-11T05:50:13.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-11T05:50:13.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-11T05:50:13.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T05:50:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:50:13.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:50:13.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-11T05:50:13.749 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T05:50:13.749 INFO:tasks.ceph:role: osd.6 2024-09-11T05:50:13.749 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:50:13.749 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:50:13.812 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:50:13.812 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:50:13.812 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:50:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:50:14.301 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-09-11T05:50:14.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-11T05:50:14.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-11T05:50:14.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-11T05:50:14.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:50:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:50:14.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-11T05:50:14.526 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T05:50:14.526 INFO:tasks.ceph:role: osd.7 2024-09-11T05:50:14.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:50:14.526 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:50:14.589 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:50:14.590 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:50:15.009 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-09-11T05:50:15.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-11T05:50:15.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-11T05:50:15.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-11T05:50:15.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:50:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:50:15.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:15.222 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:15.222+0000 7f1945ddc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T05:50:15.223 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:15.222+0000 7f1945ddc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T05:50:15.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:15.222+0000 7f1945ddc7c0 -1 bdev(0x55a9308b9800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:15.224 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:15.222+0000 7f1945ddc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T05:50:18.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T05:50:18.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:18.467 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:18.466+0000 7f152e9de7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T05:50:18.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:18.466+0000 7f152e9de7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T05:50:18.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:18.466+0000 7f152e9de7c0 -1 bdev(0x55684394f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:18.468 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:18.466+0000 7f152e9de7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T05:50:21.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T05:50:21.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:21.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:21.802+0000 7fc6c5e5e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-11T05:50:21.803 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:21.802+0000 7fc6c5e5e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-11T05:50:21.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:21.802+0000 7fc6c5e5e7c0 -1 bdev(0x55faf72c5800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:21.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:21.802+0000 7fc6c5e5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-11T05:50:24.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-11T05:50:25.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:25.104 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:25.102+0000 7f8ee17107c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-11T05:50:25.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:25.106+0000 7f8ee17107c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-11T05:50:25.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:25.106+0000 7f8ee17107c0 -1 bdev(0x55d833aff800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-11T05:50:25.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-11T05:50:25.106+0000 7f8ee17107c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-11T05:50:28.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T05:50:28.318 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T05:50:28.319 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-11T05:50:28.333 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T05:50:28.387 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T05:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T05:50:28.500 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T05:50:28.554 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T05:50:28.567 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T05:50:28.621 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T05:50:28.676 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-11T05:50:28.727 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-11T05:50:28.784 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:28.784 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T05:50:28.790 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:28.790 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-11T05:50:28.829 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T05:50:28.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T05:50:28.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.y] 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCDL+FmXGoYOhAAoqGPMJQIk3ry76tRrXp5dQ== 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.0] 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCHL+FmPWriAhAAu6+Al0WLGteysapZf0Bowg== 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.1] 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCKL+Fmh++NEhAAD1w2rN6vuhPakxLHZp8NDA== 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.2] 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCNL+Fmx5KUIhAAeYuA7hY1I6+ZHm2b9/Qjxg== 2024-09-11T05:50:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.3] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCRL+Fm/y1vBhAA9uZy2MgveDC8ZRzn4u7m1Q== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.x] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCEL+Fm2bFNARAAzYZX48EOTykrj14/bhIE7A== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.4] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCXL+FmYy9dDRAAzMY6cnXWeBqIjCD8hEk6LQ== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.5] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCaL+FmUBYBHBAAc00o7KEySPR1Ib5k69lHuw== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.6] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCdL+FmLgMCMBAATlZQ2g8vTZWE6BCTVyrJPw== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.7] 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQChL+Fm8sRfBhAAopxWtiwbpt+3BIfDulETSw== 2024-09-11T05:50:28.876 INFO:teuthology.orchestra.run.smithi165.stdout:[client.0] 2024-09-11T05:50:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCEL+Fm5O6IBBAAI0hik/GlfHSXtISnzTJ3qg== 2024-09-11T05:50:28.877 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-09-11T05:50:28.877 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCEL+Fmn6ZDBxAAZ893GkqyFPOELfe8T9TlTA== 2024-09-11T05:50:28.882 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2024-09-11T05:50:28.882 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCDL+FmXGoYOhAAoqGPMJQIk3ry76tRrXp5dQ== 2024-09-11T05:50:28.882 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-09-11T05:50:28.882 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCHL+FmPWriAhAAu6+Al0WLGteysapZf0Bowg== 2024-09-11T05:50:28.882 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCKL+Fmh++NEhAAD1w2rN6vuhPakxLHZp8NDA== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCNL+Fmx5KUIhAAeYuA7hY1I6+ZHm2b9/Qjxg== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCRL+Fm/y1vBhAA9uZy2MgveDC8ZRzn4u7m1Q== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCEL+Fm2bFNARAAzYZX48EOTykrj14/bhIE7A== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCXL+FmYy9dDRAAzMY6cnXWeBqIjCD8hEk6LQ== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCaL+FmUBYBHBAAc00o7KEySPR1Ib5k69lHuw== 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2024-09-11T05:50:28.883 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCdL+FmLgMCMBAATlZQ2g8vTZWE6BCTVyrJPw== 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQChL+Fm8sRfBhAAopxWtiwbpt+3BIfDulETSw== 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCEL+Fm5O6IBBAAI0hik/GlfHSXtISnzTJ3qg== 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-09-11T05:50:28.884 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCEL+Fmn6ZDBxAAZ893GkqyFPOELfe8T9TlTA== 2024-09-11T05:50:28.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:50:28.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:50:28.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:28.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:50:29.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:50:29.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:50:29.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:50:29.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:50:29.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:50:29.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:50:29.938 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T05:50:29.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T05:50:29.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:50:30.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T05:50:30.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T05:50:30.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:50:30.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T05:50:30.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T05:50:30.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:50:30.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T05:50:30.350 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:30.352 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:50:30.397 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T05:50:30.397 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T05:50:30.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T05:50:30.400 INFO:tasks.ceph.mon.a:Started 2024-09-11T05:50:30.401 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T05:50:30.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T05:50:30.403 INFO:tasks.ceph.mon.c:Started 2024-09-11T05:50:30.404 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:50:30.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:50:30.442 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:50:30.442 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T05:50:30.443 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-11T05:50:30.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-11T05:50:30.445 INFO:tasks.ceph.mgr.y:Started 2024-09-11T05:50:30.445 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T05:50:30.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T05:50:30.447 INFO:tasks.ceph.mgr.x:Started 2024-09-11T05:50:30.447 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T05:50:30.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T05:50:35.921 INFO:teuthology.orchestra.run.smithi165.stderr:adjusted tunables profile to default 2024-09-11T05:50:35.941 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T05:50:35.941 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T05:50:35.941 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:35.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T05:50:35.952 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:35.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T05:50:36.008 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:36.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T05:50:36.067 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-11T05:50:36.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T05:50:36.067 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:36.066+0000 7fc4f1391180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T05:50:36.125 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:36.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T05:50:36.141 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:36.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T05:50:36.197 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:36.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-11T05:50:36.251 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-11T05:50:36.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-11T05:50:36.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 2fdc5357-7501-4582-b301-f5eede8a7bc0 0 2024-09-11T05:50:36.562 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:36.558+0000 7fc4f1391180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T05:50:36.567 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-09-11T05:50:36.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 78f30980-40a1-46fd-bc1a-a656b32210a9 1 2024-09-11T05:50:36.756 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:36.754+0000 7f3702d94180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T05:50:36.760 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:36.758+0000 7fc4f1391180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T05:50:36.787 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-09-11T05:50:36.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 2fab9ebc-408e-4bc1-b3ad-385d0e396f6b 2 2024-09-11T05:50:37.016 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-09-11T05:50:37.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new d3115ede-8d17-4a27-8629-40b19e3c7ee6 3 2024-09-11T05:50:37.237 INFO:teuthology.orchestra.run.smithi172.stdout:3 2024-09-11T05:50:37.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 7fa88502-3ffd-4aac-924b-0d6c65737ac6 4 2024-09-11T05:50:37.293 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:37.294+0000 7f3702d94180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T05:50:37.310 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.306+0000 7fc4f1391180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T05:50:37.363 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.362+0000 7fc4f1391180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T05:50:37.416 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.414+0000 7fc4f1391180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T05:50:37.458 INFO:teuthology.orchestra.run.smithi172.stdout:4 2024-09-11T05:50:37.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 03238cca-4d2b-412d-9fcd-f489cb4a8455 5 2024-09-11T05:50:37.493 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.490+0000 7fc4f1391180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T05:50:37.523 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:37.522+0000 7f3702d94180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T05:50:37.544 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.542+0000 7fc4f1391180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T05:50:37.620 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.618+0000 7fc4f1391180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T05:50:37.673 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.670+0000 7fc4f1391180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T05:50:37.733 INFO:teuthology.orchestra.run.smithi172.stdout:5 2024-09-11T05:50:37.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 6c46c2d3-7dd1-4b3a-ad76-391c6e624b40 6 2024-09-11T05:50:37.825 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.822+0000 7fc4f1391180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T05:50:37.887 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:37.886+0000 7fc4f1391180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T05:50:37.989 INFO:teuthology.orchestra.run.smithi172.stdout:6 2024-09-11T05:50:38.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new f849014a-7b82-4361-8a23-3b7a0be5f5c0 7 2024-09-11T05:50:38.115 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.114+0000 7f3702d94180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T05:50:38.173 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.174+0000 7f3702d94180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T05:50:38.209 INFO:teuthology.orchestra.run.smithi172.stdout:7 2024-09-11T05:50:38.225 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T05:50:38.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T05:50:38.227 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.226+0000 7f3702d94180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T05:50:38.228 INFO:tasks.ceph.osd.0:Started 2024-09-11T05:50:38.228 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T05:50:38.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T05:50:38.231 INFO:tasks.ceph.osd.1:Started 2024-09-11T05:50:38.231 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T05:50:38.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T05:50:38.233 INFO:tasks.ceph.osd.2:Started 2024-09-11T05:50:38.233 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T05:50:38.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T05:50:38.235 INFO:tasks.ceph.osd.3:Started 2024-09-11T05:50:38.235 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T05:50:38.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T05:50:38.237 INFO:tasks.ceph.osd.4:Started 2024-09-11T05:50:38.238 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T05:50:38.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T05:50:38.239 INFO:tasks.ceph.osd.5:Started 2024-09-11T05:50:38.239 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-11T05:50:38.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-11T05:50:38.241 INFO:tasks.ceph.osd.6:Started 2024-09-11T05:50:38.241 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-11T05:50:38.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-11T05:50:38.243 INFO:tasks.ceph.osd.7:Started 2024-09-11T05:50:38.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:38.316 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.314+0000 7f3702d94180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T05:50:38.370 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.370+0000 7f3702d94180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T05:50:38.447 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.446+0000 7f3702d94180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T05:50:38.466 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:50:38.467 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":10,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:38.206686+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:38.482 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T05:50:38.483 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T05:50:38.500 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.498+0000 7f3702d94180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T05:50:38.545 INFO:tasks.ceph.mgr.y.smithi165.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T05:50:38.545 INFO:tasks.ceph.mgr.y.smithi165.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T05:50:38.545 INFO:tasks.ceph.mgr.y.smithi165.stderr: from numpy import show_config as show_numpy_config 2024-09-11T05:50:38.625 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:38.622+0000 7fc4f1391180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T05:50:38.650 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.650+0000 7f3702d94180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T05:50:38.712 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:38.710+0000 7f3702d94180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T05:50:38.787 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:38.782+0000 7fc4f1391180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:50:38.950 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:38.946+0000 7fc4f1391180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T05:50:39.019 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:39.018+0000 7fc4f1391180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T05:50:39.166 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:39.162+0000 7fc4f1391180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:50:39.288 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:39.323 INFO:tasks.ceph.mgr.x.smithi172.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T05:50:39.323 INFO:tasks.ceph.mgr.x.smithi172.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T05:50:39.323 INFO:tasks.ceph.mgr.x.smithi172.stderr: from numpy import show_config as show_numpy_config 2024-09-11T05:50:39.380 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:39.378+0000 7f3702d94180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T05:50:39.468 INFO:teuthology.misc.health.smithi165.stdout: 2024-09-11T05:50:39.469 INFO:teuthology.misc.health.smithi165.stdout:{"epoch":10,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:38.206686+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:39.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-11T05:50:39.532 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:39.530+0000 7f3702d94180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:50:39.673 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:50:39.674+0000 7f50ef0b07c0 -1 Falling back to public interface 2024-09-11T05:50:39.694 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:39.694+0000 7f3702d94180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T05:50:39.757 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:39.758+0000 7f3702d94180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T05:50:39.771 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:39.770+0000 7fc4f1391180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T05:50:39.847 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:39.846+0000 7fc4f1391180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T05:50:39.881 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:50:39.882+0000 7f1d3fc737c0 -1 Falling back to public interface 2024-09-11T05:50:39.897 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:39.898+0000 7f3702d94180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:50:40.029 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:40.026+0000 7fc4f1391180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T05:50:40.102 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:40.098+0000 7fc4f1391180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T05:50:40.117 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:50:40.118+0000 7f36bbcfe7c0 -1 Falling back to public interface 2024-09-11T05:50:40.152 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:50:40.150+0000 7f916218b7c0 -1 Falling back to public interface 2024-09-11T05:50:40.200 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:50:40.198+0000 7fa274b117c0 -1 Falling back to public interface 2024-09-11T05:50:40.203 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:50:40.202+0000 7f0cdbdf97c0 -1 Falling back to public interface 2024-09-11T05:50:40.364 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:50:40.362+0000 7f26db3b67c0 -1 Falling back to public interface 2024-09-11T05:50:40.374 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:50:40.374+0000 7fa6740cb7c0 -1 Falling back to public interface 2024-09-11T05:50:40.397 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:40.394+0000 7fc4f1391180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T05:50:40.459 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:40.458+0000 7fc4f1391180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T05:50:40.518 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:40.518+0000 7f3702d94180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T05:50:40.595 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:40.594+0000 7f3702d94180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T05:50:40.778 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:40.778+0000 7f3702d94180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T05:50:40.853 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:40.854+0000 7f3702d94180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T05:50:41.154 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:41.154+0000 7f3702d94180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T05:50:41.216 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-09-11T05:50:41.214+0000 7f3702d94180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T05:50:41.498 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:41.494+0000 7fc497ec8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T05:50:41.503 INFO:tasks.ceph.mgr.y.smithi165.stderr:2024-09-11T05:50:41.502+0000 7fc497ec8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T05:50:44.402 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:50:44.402+0000 7fa6740cb7c0 -1 osd.6 0 log_to_monitors true 2024-09-11T05:50:44.411 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:50:44.410+0000 7f50ef0b07c0 -1 osd.4 0 log_to_monitors true 2024-09-11T05:50:44.555 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:50:44.554+0000 7f0cdbdf97c0 -1 osd.0 0 log_to_monitors true 2024-09-11T05:50:44.578 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:50:44.574+0000 7fa274b117c0 -1 osd.2 0 log_to_monitors true 2024-09-11T05:50:44.580 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:50:44.582+0000 7f36bbcfe7c0 -1 osd.5 0 log_to_monitors true 2024-09-11T05:50:44.586 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:50:44.586+0000 7f1d3fc737c0 -1 osd.7 0 log_to_monitors true 2024-09-11T05:50:44.708 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:50:44.706+0000 7f916218b7c0 -1 osd.1 0 log_to_monitors true 2024-09-11T05:50:45.154 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:50:45.150+0000 7f26db3b67c0 -1 osd.3 0 log_to_monitors true 2024-09-11T05:50:46.289 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:46.626 INFO:teuthology.misc.health.smithi165.stdout: 2024-09-11T05:50:46.626 INFO:teuthology.misc.health.smithi165.stdout:{"epoch":12,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:45.849927+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:46.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-11T05:50:48.036 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:50:48.034+0000 7fa270297640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T05:50:48.047 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:50:48.042+0000 7f915d911640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T05:50:48.070 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:50:48.066+0000 7fa2688de640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.078 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:50:48.074+0000 7f0cd757f640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T05:50:48.084 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:50:48.082+0000 7f9155f58640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.127 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:50:48.126+0000 7f0ccfbc6640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.198 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:50:48.198+0000 7fa670886640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T05:50:48.213 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:50:48.214+0000 7f50eb86b640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T05:50:48.226 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:50:48.226+0000 7fa667e98640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.241 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:50:48.242+0000 7f50e2e7d640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.275 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:50:48.274+0000 7f1d3c42e640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T05:50:48.309 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:50:48.310+0000 7f1d33a40640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.337 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:50:48.338+0000 7f36b7484640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T05:50:48.365 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:50:48.366+0000 7f36afacb640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:48.414 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:50:48.410+0000 7f26d6b3c640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T05:50:48.456 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:50:48.454+0000 7f26cf183640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:50:53.452 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:53.818 INFO:teuthology.misc.health.smithi165.stdout: 2024-09-11T05:50:53.819 INFO:teuthology.misc.health.smithi165.stdout:{"epoch":18,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:53.115621+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:53.837 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-11T05:50:53.837 INFO:tasks.ceph:Creating RBD pool 2024-09-11T05:50:53.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T05:50:55.139 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'rbd' created 2024-09-11T05:50:55.162 DEBUG:teuthology.orchestra.run.smithi165:> rbd --cluster ceph pool init rbd 2024-09-11T05:50:58.195 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T05:50:58.196 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T05:50:58.196 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T05:50:58.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T05:50:58.197 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T05:50:58.549 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:50:58.567 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1965965311},{"type":"v1","addr":"172.21.15.165:6833","nonce":1965965311}]},"active_addr":"172.21.15.165:6833/1965965311","active_change":"2024-09-11T05:50:40.481864+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4117,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":194947513}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1205647348}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1416957887}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":2004163985}]}]} 2024-09-11T05:50:58.570 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T05:50:58.570 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T05:50:58.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:58.864 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:50:58.864 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":22,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:58.154584+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:50:54.170388+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:58.881 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T05:50:58.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:50:59.168 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:50:59.168 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":22,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:50:58.154584+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:50:38.206686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:50:54.170388+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:50:59.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T05:50:59.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T05:50:59.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T05:50:59.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T05:50:59.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T05:50:59.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T05:50:59.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-11T05:50:59.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-11T05:50:59.439 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:50:59.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T05:50:59.451 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:50:59.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:50:59.457 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:50:59.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:50:59.460 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:50:59.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:50:59.461 INFO:teuthology.orchestra.run.smithi165.stdout:60129542148 2024-09-11T05:50:59.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:50:59.465 INFO:teuthology.orchestra.run.smithi165.stdout:60129542148 2024-09-11T05:50:59.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:50:59.472 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:50:59.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T05:50:59.476 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:50:59.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:50:59.935 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:50:59.942 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:50:59.962 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.6 2024-09-11T05:50:59.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-09-11T05:50:59.982 INFO:teuthology.orchestra.run.smithi165.stdout:60129542147 2024-09-11T05:50:59.996 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:50:59.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.2 2024-09-11T05:51:00.001 INFO:teuthology.orchestra.run.smithi165.stdout:60129542147 2024-09-11T05:51:00.003 INFO:teuthology.orchestra.run.smithi165.stdout:60129542148 2024-09-11T05:51:00.020 INFO:teuthology.orchestra.run.smithi165.stdout:60129542148 2024-09-11T05:51:00.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-09-11T05:51:00.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.0 2024-09-11T05:51:00.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.1 2024-09-11T05:51:00.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.3 2024-09-11T05:51:00.056 INFO:teuthology.orchestra.run.smithi165.stdout:60129542148 2024-09-11T05:51:00.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.5 2024-09-11T05:51:00.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T05:51:00.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:51:01.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:51:01.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T05:51:01.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:51:01.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:51:01.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:51:01.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:51:01.469 INFO:teuthology.orchestra.run.smithi165.stdout:60129542151 2024-09-11T05:51:01.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.6 2024-09-11T05:51:01.491 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.514 INFO:teuthology.orchestra.run.smithi165.stdout:60129542151 2024-09-11T05:51:01.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.4 2024-09-11T05:51:01.539 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.541 INFO:teuthology.orchestra.run.smithi165.stdout:60129542151 2024-09-11T05:51:01.545 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:51:01.556 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:51:01.562 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.0 2024-09-11T05:51:01.562 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.2 2024-09-11T05:51:01.576 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.581 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:51:01.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.7 2024-09-11T05:51:01.584 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.589 INFO:teuthology.orchestra.run.smithi165.stdout:60129542149 2024-09-11T05:51:01.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-09-11T05:51:01.600 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.1 2024-09-11T05:51:01.608 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.609 INFO:teuthology.orchestra.run.smithi165.stdout:60129542150 2024-09-11T05:51:01.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.5 2024-09-11T05:51:01.626 DEBUG:teuthology.parallel:result is None 2024-09-11T05:51:01.626 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:51:01.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:01.933 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:01.934 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:01.950 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T05:51:00.493451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":219192,"kb_used_data":2928,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":2998272,"data_stored":1795734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.356195"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.466603+0000","last_change":"2024-09-11T05:50:58.466603+0000","last_active":"2024-09-11T05:50:58.466603+0000","last_peered":"2024-09-11T05:50:58.466603+0000","last_clean":"2024-09-11T05:50:58.466603+0000","last_became_active":"2024-09-11T05:50:56.156045+0000","last_became_peered":"2024-09-11T05:50:56.156045+0000","last_unstale":"2024-09-11T05:50:58.466603+0000","last_undegraded":"2024-09-11T05:50:58.466603+0000","last_fullsized":"2024-09-11T05:50:58.466603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:39.996257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011063640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177276+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:50:58.177276+0000","last_peered":"2024-09-11T05:50:58.177276+0000","last_clean":"2024-09-11T05:50:58.177276+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:50:58.177276+0000","last_undegraded":"2024-09-11T05:50:58.177276+0000","last_fullsized":"2024-09-11T05:50:58.177276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467242+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:50:58.467242+0000","last_peered":"2024-09-11T05:50:58.467242+0000","last_clean":"2024-09-11T05:50:58.467242+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:50:58.467242+0000","last_undegraded":"2024-09-11T05:50:58.467242+0000","last_fullsized":"2024-09-11T05:50:58.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177342+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:50:58.177342+0000","last_peered":"2024-09-11T05:50:58.177342+0000","last_clean":"2024-09-11T05:50:58.177342+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:50:58.177342+0000","last_undegraded":"2024-09-11T05:50:58.177342+0000","last_fullsized":"2024-09-11T05:50:58.177342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.176571+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:50:58.176571+0000","last_peered":"2024-09-11T05:50:58.176571+0000","last_clean":"2024-09-11T05:50:58.176571+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:50:58.176571+0000","last_undegraded":"2024-09-11T05:50:58.176571+0000","last_fullsized":"2024-09-11T05:50:58.176571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.167606+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:50:58.167606+0000","last_peered":"2024-09-11T05:50:58.167606+0000","last_clean":"2024-09-11T05:50:58.167606+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:50:58.167606+0000","last_undegraded":"2024-09-11T05:50:58.167606+0000","last_fullsized":"2024-09-11T05:50:58.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467120+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:50:58.467120+0000","last_peered":"2024-09-11T05:50:58.467120+0000","last_clean":"2024-09-11T05:50:58.467120+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:50:58.467120+0000","last_undegraded":"2024-09-11T05:50:58.467120+0000","last_fullsized":"2024-09-11T05:50:58.467120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.173989+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:50:58.173989+0000","last_peered":"2024-09-11T05:50:58.173989+0000","last_clean":"2024-09-11T05:50:58.173989+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:50:58.173989+0000","last_undegraded":"2024-09-11T05:50:58.173989+0000","last_fullsized":"2024-09-11T05:50:58.173989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.465732+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:50:58.465732+0000","last_peered":"2024-09-11T05:50:58.465732+0000","last_clean":"2024-09-11T05:50:58.465732+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:50:58.465732+0000","last_undegraded":"2024-09-11T05:50:58.465732+0000","last_fullsized":"2024-09-11T05:50:58.465732+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:01.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:02.227 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:02.227 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:02.244 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T05:51:00.493451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":219192,"kb_used_data":2928,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":2998272,"data_stored":1795734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.356195"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.466603+0000","last_change":"2024-09-11T05:50:58.466603+0000","last_active":"2024-09-11T05:50:58.466603+0000","last_peered":"2024-09-11T05:50:58.466603+0000","last_clean":"2024-09-11T05:50:58.466603+0000","last_became_active":"2024-09-11T05:50:56.156045+0000","last_became_peered":"2024-09-11T05:50:56.156045+0000","last_unstale":"2024-09-11T05:50:58.466603+0000","last_undegraded":"2024-09-11T05:50:58.466603+0000","last_fullsized":"2024-09-11T05:50:58.466603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:39.996257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011063640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177276+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:50:58.177276+0000","last_peered":"2024-09-11T05:50:58.177276+0000","last_clean":"2024-09-11T05:50:58.177276+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:50:58.177276+0000","last_undegraded":"2024-09-11T05:50:58.177276+0000","last_fullsized":"2024-09-11T05:50:58.177276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467242+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:50:58.467242+0000","last_peered":"2024-09-11T05:50:58.467242+0000","last_clean":"2024-09-11T05:50:58.467242+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:50:58.467242+0000","last_undegraded":"2024-09-11T05:50:58.467242+0000","last_fullsized":"2024-09-11T05:50:58.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177342+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:50:58.177342+0000","last_peered":"2024-09-11T05:50:58.177342+0000","last_clean":"2024-09-11T05:50:58.177342+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:50:58.177342+0000","last_undegraded":"2024-09-11T05:50:58.177342+0000","last_fullsized":"2024-09-11T05:50:58.177342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.176571+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:50:58.176571+0000","last_peered":"2024-09-11T05:50:58.176571+0000","last_clean":"2024-09-11T05:50:58.176571+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:50:58.176571+0000","last_undegraded":"2024-09-11T05:50:58.176571+0000","last_fullsized":"2024-09-11T05:50:58.176571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.167606+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:50:58.167606+0000","last_peered":"2024-09-11T05:50:58.167606+0000","last_clean":"2024-09-11T05:50:58.167606+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:50:58.167606+0000","last_undegraded":"2024-09-11T05:50:58.167606+0000","last_fullsized":"2024-09-11T05:50:58.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467120+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:50:58.467120+0000","last_peered":"2024-09-11T05:50:58.467120+0000","last_clean":"2024-09-11T05:50:58.467120+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:50:58.467120+0000","last_undegraded":"2024-09-11T05:50:58.467120+0000","last_fullsized":"2024-09-11T05:50:58.467120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.173989+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:50:58.173989+0000","last_peered":"2024-09-11T05:50:58.173989+0000","last_clean":"2024-09-11T05:50:58.173989+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:50:58.173989+0000","last_undegraded":"2024-09-11T05:50:58.173989+0000","last_fullsized":"2024-09-11T05:50:58.173989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.465732+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:50:58.465732+0000","last_peered":"2024-09-11T05:50:58.465732+0000","last_clean":"2024-09-11T05:50:58.465732+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:50:58.465732+0000","last_undegraded":"2024-09-11T05:50:58.465732+0000","last_fullsized":"2024-09-11T05:50:58.465732+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:02.245 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:51:02.245 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T05:51:02.245 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T05:51:02.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T05:51:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:02.557 INFO:teuthology.orchestra.run.smithi165.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T05:51:02.575 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T05:51:02.575 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-11T05:51:02.586 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T05:51:02.586 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}} 2024-09-11T05:51:02.586 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-11T05:51:02.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-11T05:51:02.888 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-11T05:51:02.889 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:51:02.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:03.157 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:03.157 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:03.174 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T05:51:02.494096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":219192,"kb_used_data":2928,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":2998272,"data_stored":1795734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.354361"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.466603+0000","last_change":"2024-09-11T05:50:58.466603+0000","last_active":"2024-09-11T05:50:58.466603+0000","last_peered":"2024-09-11T05:50:58.466603+0000","last_clean":"2024-09-11T05:50:58.466603+0000","last_became_active":"2024-09-11T05:50:56.156045+0000","last_became_peered":"2024-09-11T05:50:56.156045+0000","last_unstale":"2024-09-11T05:50:58.466603+0000","last_undegraded":"2024-09-11T05:50:58.466603+0000","last_fullsized":"2024-09-11T05:50:58.466603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:39.996257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011063640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177276+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:50:58.177276+0000","last_peered":"2024-09-11T05:50:58.177276+0000","last_clean":"2024-09-11T05:50:58.177276+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:50:58.177276+0000","last_undegraded":"2024-09-11T05:50:58.177276+0000","last_fullsized":"2024-09-11T05:50:58.177276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467242+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:50:58.467242+0000","last_peered":"2024-09-11T05:50:58.467242+0000","last_clean":"2024-09-11T05:50:58.467242+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:50:58.467242+0000","last_undegraded":"2024-09-11T05:50:58.467242+0000","last_fullsized":"2024-09-11T05:50:58.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177342+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:50:58.177342+0000","last_peered":"2024-09-11T05:50:58.177342+0000","last_clean":"2024-09-11T05:50:58.177342+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:50:58.177342+0000","last_undegraded":"2024-09-11T05:50:58.177342+0000","last_fullsized":"2024-09-11T05:50:58.177342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.176571+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:50:58.176571+0000","last_peered":"2024-09-11T05:50:58.176571+0000","last_clean":"2024-09-11T05:50:58.176571+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:50:58.176571+0000","last_undegraded":"2024-09-11T05:50:58.176571+0000","last_fullsized":"2024-09-11T05:50:58.176571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.167606+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:50:58.167606+0000","last_peered":"2024-09-11T05:50:58.167606+0000","last_clean":"2024-09-11T05:50:58.167606+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:50:58.167606+0000","last_undegraded":"2024-09-11T05:50:58.167606+0000","last_fullsized":"2024-09-11T05:50:58.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467120+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:50:58.467120+0000","last_peered":"2024-09-11T05:50:58.467120+0000","last_clean":"2024-09-11T05:50:58.467120+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:50:58.467120+0000","last_undegraded":"2024-09-11T05:50:58.467120+0000","last_fullsized":"2024-09-11T05:50:58.467120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.173989+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:50:58.173989+0000","last_peered":"2024-09-11T05:50:58.173989+0000","last_clean":"2024-09-11T05:50:58.173989+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:50:58.173989+0000","last_undegraded":"2024-09-11T05:50:58.173989+0000","last_fullsized":"2024-09-11T05:50:58.173989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.465732+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:50:58.465732+0000","last_peered":"2024-09-11T05:50:58.465732+0000","last_clean":"2024-09-11T05:50:58.465732+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:50:58.465732+0000","last_undegraded":"2024-09-11T05:50:58.465732+0000","last_fullsized":"2024-09-11T05:50:58.465732+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:03.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:03.451 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:03.451 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:03.467 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T05:51:02.494096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":219192,"kb_used_data":2928,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":2998272,"data_stored":1795734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.354361"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.466603+0000","last_change":"2024-09-11T05:50:58.466603+0000","last_active":"2024-09-11T05:50:58.466603+0000","last_peered":"2024-09-11T05:50:58.466603+0000","last_clean":"2024-09-11T05:50:58.466603+0000","last_became_active":"2024-09-11T05:50:56.156045+0000","last_became_peered":"2024-09-11T05:50:56.156045+0000","last_unstale":"2024-09-11T05:50:58.466603+0000","last_undegraded":"2024-09-11T05:50:58.466603+0000","last_fullsized":"2024-09-11T05:50:58.466603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:06:39.996257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011063640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177276+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:50:58.177276+0000","last_peered":"2024-09-11T05:50:58.177276+0000","last_clean":"2024-09-11T05:50:58.177276+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:50:58.177276+0000","last_undegraded":"2024-09-11T05:50:58.177276+0000","last_fullsized":"2024-09-11T05:50:58.177276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467242+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:50:58.467242+0000","last_peered":"2024-09-11T05:50:58.467242+0000","last_clean":"2024-09-11T05:50:58.467242+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:50:58.467242+0000","last_undegraded":"2024-09-11T05:50:58.467242+0000","last_fullsized":"2024-09-11T05:50:58.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.177342+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:50:58.177342+0000","last_peered":"2024-09-11T05:50:58.177342+0000","last_clean":"2024-09-11T05:50:58.177342+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:50:58.177342+0000","last_undegraded":"2024-09-11T05:50:58.177342+0000","last_fullsized":"2024-09-11T05:50:58.177342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.176571+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:50:58.176571+0000","last_peered":"2024-09-11T05:50:58.176571+0000","last_clean":"2024-09-11T05:50:58.176571+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:50:58.176571+0000","last_undegraded":"2024-09-11T05:50:58.176571+0000","last_fullsized":"2024-09-11T05:50:58.176571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.167606+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:50:58.167606+0000","last_peered":"2024-09-11T05:50:58.167606+0000","last_clean":"2024-09-11T05:50:58.167606+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:50:58.167606+0000","last_undegraded":"2024-09-11T05:50:58.167606+0000","last_fullsized":"2024-09-11T05:50:58.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.467120+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:50:58.467120+0000","last_peered":"2024-09-11T05:50:58.467120+0000","last_clean":"2024-09-11T05:50:58.467120+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:50:58.467120+0000","last_undegraded":"2024-09-11T05:50:58.467120+0000","last_fullsized":"2024-09-11T05:50:58.467120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.173989+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:50:58.173989+0000","last_peered":"2024-09-11T05:50:58.173989+0000","last_clean":"2024-09-11T05:50:58.173989+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:50:58.173989+0000","last_undegraded":"2024-09-11T05:50:58.173989+0000","last_fullsized":"2024-09-11T05:50:58.173989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T05:50:58.465732+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:50:58.465732+0000","last_peered":"2024-09-11T05:50:58.465732+0000","last_clean":"2024-09-11T05:50:58.465732+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:50:58.465732+0000","last_undegraded":"2024-09-11T05:50:58.465732+0000","last_fullsized":"2024-09-11T05:50:58.465732+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":667238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:03.468 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:51:03.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:epoch 22 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:fsid d5628057-c743-43f5-a51d-1915a16ccfdd 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:created 2024-09-11T05:50:35.802404+0000 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:modified 2024-09-11T05:50:58.154584+0000 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-11T05:51:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:crush_version 5 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:full_ratio 0.95 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:backfillfull_ratio 0.9 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:nearfull_ratio 0.85 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:require_min_compat_client luminous 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:min_compat_client jewel 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:require_osd_release squid 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:stretch_mode_enabled false 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 22 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.99 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout: removed_snaps_queue [2~1] 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:max_osd 8 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6815/4049823921,v1:172.21.15.165:6817/4049823921] [v2:172.21.15.165:6818/4049823921,v1:172.21.15.165:6819/4049823921] exists,up 2fdc5357-7501-4582-b301-f5eede8a7bc0 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/2461301418,v1:172.21.15.165:6801/2461301418] [v2:172.21.15.165:6802/2461301418,v1:172.21.15.165:6803/2461301418] exists,up 78f30980-40a1-46fd-bc1a-a656b32210a9 2024-09-11T05:51:03.768 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/3166670912,v1:172.21.15.165:6809/3166670912] [v2:172.21.15.165:6810/3166670912,v1:172.21.15.165:6811/3166670912] exists,up 2fab9ebc-408e-4bc1-b3ad-385d0e396f6b 2024-09-11T05:51:03.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/3322140846,v1:172.21.15.165:6825/3322140846] [v2:172.21.15.165:6826/3322140846,v1:172.21.15.165:6827/3322140846] exists,up d3115ede-8d17-4a27-8629-40b19e3c7ee6 2024-09-11T05:51:03.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/347599006,v1:172.21.15.172:6801/347599006] [v2:172.21.15.172:6802/347599006,v1:172.21.15.172:6803/347599006] exists,up 7fa88502-3ffd-4aac-924b-0d6c65737ac6 2024-09-11T05:51:03.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/4259095655,v1:172.21.15.172:6817/4259095655] [v2:172.21.15.172:6818/4259095655,v1:172.21.15.172:6819/4259095655] exists,up 03238cca-4d2b-412d-9fcd-f489cb4a8455 2024-09-11T05:51:03.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/2366978310,v1:172.21.15.172:6825/2366978310] [v2:172.21.15.172:6826/2366978310,v1:172.21.15.172:6827/2366978310] exists,up 6c46c2d3-7dd1-4b3a-ad76-391c6e624b40 2024-09-11T05:51:03.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/2576837456,v1:172.21.15.172:6809/2576837456] [v2:172.21.15.172:6810/2576837456,v1:172.21.15.172:6811/2576837456] exists,up f849014a-7b82-4361-8a23-3b7a0be5f5c0 2024-09-11T05:51:03.786 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6815/4049823921,v1:172.21.15.165:6817/4049823921] [v2:172.21.15.165:6818/4049823921,v1:172.21.15.165:6819/4049823921] exists,up 2fdc5357-7501-4582-b301-f5eede8a7bc0', 'osd.1 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/2461301418,v1:172.21.15.165:6801/2461301418] [v2:172.21.15.165:6802/2461301418,v1:172.21.15.165:6803/2461301418] exists,up 78f30980-40a1-46fd-bc1a-a656b32210a9', 'osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/3166670912,v1:172.21.15.165:6809/3166670912] [v2:172.21.15.165:6810/3166670912,v1:172.21.15.165:6811/3166670912] exists,up 2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/3322140846,v1:172.21.15.165:6825/3322140846] [v2:172.21.15.165:6826/3322140846,v1:172.21.15.165:6827/3322140846] exists,up d3115ede-8d17-4a27-8629-40b19e3c7ee6', 'osd.4 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/347599006,v1:172.21.15.172:6801/347599006] [v2:172.21.15.172:6802/347599006,v1:172.21.15.172:6803/347599006] exists,up 7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'osd.5 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/4259095655,v1:172.21.15.172:6817/4259095655] [v2:172.21.15.172:6818/4259095655,v1:172.21.15.172:6819/4259095655] exists,up 03238cca-4d2b-412d-9fcd-f489cb4a8455', 'osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/2366978310,v1:172.21.15.172:6825/2366978310] [v2:172.21.15.172:6826/2366978310,v1:172.21.15.172:6827/2366978310] exists,up 6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'osd.7 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/2576837456,v1:172.21.15.172:6809/2576837456] [v2:172.21.15.172:6810/2576837456,v1:172.21.15.172:6811/2576837456] exists,up f849014a-7b82-4361-8a23-3b7a0be5f5c0'] 2024-09-11T05:51:03.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-11T05:51:03.898 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:03.898 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "mon.a", 2024-09-11T05:51:03.898 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster": "ceph", 2024-09-11T05:51:03.898 INFO:teuthology.orchestra.run.smithi165.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-11T05:51:03.898 INFO:teuthology.orchestra.run.smithi165.stdout: "admin_socket_mode": "", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_client_required": "cephx, none", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_cluster_required": "cephx", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_debug": "true", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_service_required": "cephx", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "auth_supported": "cephx", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_aio": "true", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_aio_poll_ms": "250", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_aio_reap_max": "16", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_async_discard": "false", 2024-09-11T05:51:03.899 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_block_size": "4096", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_debug_aio": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_debug_inflight_ios": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_enable_discard": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_flock_retry": "3", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_inject_crash": "0", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_ioring": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_ioring_hipri": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-11T05:51:03.900 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bdev_type": "", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_alloc_size": "1048576", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_allocator": "hybrid", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_buffered_io": "true", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_check_for_zeros": "false", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_compact_log_sync": "false", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-11T05:51:03.901 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_max_log_runway": "4194304", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_max_prefetch": "1048576", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_min_flush_size": "524288", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_min_log_runway": "1048576", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_replay_recovery": "false", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_sync_write": "false", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_allocation_from_file": "true", 2024-09-11T05:51:03.902 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_allocator": "hybrid", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bdev_label_multi": "true", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_create": "true", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_db_create": "false", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_db_path": "", 2024-09-11T05:51:03.903 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_db_size": "0", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_path": "", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_preallocate_file": "false", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_size": "107374182400", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_wal_create": "false", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_wal_path": "", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_block_wal_size": "100663296", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bluefs": "true", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-11T05:51:03.904 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_autotune": "true", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_size": "0", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_type": "2q", 2024-09-11T05:51:03.905 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_clone_cow": "true", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_mode": "none", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_csum_type": "crc32c", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_freelist": "false", 2024-09-11T05:51:03.906 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_small_allocations": "0", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_default_buffered_read": "true", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_default_buffered_write": "false", 2024-09-11T05:51:03.907 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fail_eio": "false", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-11T05:51:03.908 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_mount": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_umount": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-11T05:51:03.909 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_ignore_data_csum": "false", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_log_op_age": "5.000000", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_alloc_size": "0", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_blob_size": "0", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_max_deferred_txc": "32", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_min_alloc_size": "0", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-11T05:51:03.910 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_nid_prealloc": "1024", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_retry_disk_reads": "3", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_rocksdb_cf": "true", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-11T05:51:03.911 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_spdk_mem": "512", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_tracing": "false", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-11T05:51:03.912 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_zero_block_detection": "false", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-11T05:51:03.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_cluster_require_signatures": "false", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_cluster_require_version": "2", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_require_signatures": "false", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_require_version": "2", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_service_require_signatures": "false", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_service_require_version": "2", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "cephx_sign_messages": "true", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "chdir": "", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_acl_type": "", 2024-09-11T05:51:03.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_asio_thread_count": "2", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_cache_mid": "0.750000", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_cache_size": "16384", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_caps_release_delay": "5", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_check_pool_perm": "true", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_debug_force_sync_read": "false", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_debug_getattr_caps": "false", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_debug_inject_features": "", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_debug_inject_tick_delay": "0", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_die_on_failed_remount": "false", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_dirsize_rbytes": "true", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_force_lazyio": "false", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_fs": "", 2024-09-11T05:51:03.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_inject_release_failure": "false", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_max_inline_size": "4096", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_mds_namespace": "", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_metadata": "", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_mount_gid": "-1", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_mount_timeout": "300", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_mount_uid": "-1", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_mountpoint": "/", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_notify_timeout": "10", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc": "true", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc_max_dirty": "104857600", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc_max_objects": "1000", 2024-09-11T05:51:03.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc_size": "209715200", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_oc_target_dirty": "8388608", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_permissions": "true", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_quota": "true", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_quota_df": "true", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_readahead_max_bytes": "0", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_readahead_max_periods": "4", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_readahead_min": "131072", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_reconnect_stale": "false", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_shutdown_timeout": "30", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_snapdir": ".snap", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_tick_interval": "1", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_trace": "", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_try_dentry_invalidate": "false", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_use_faked_inos": "false", 2024-09-11T05:51:03.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_use_random_mds": "false", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_graylog": "false", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_graylog_port": "12201", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_monitors": "default=true", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_syslog": "false", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "clog_to_syslog_level": "info", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "-", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_network": "", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_network_interface": "", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "compressor_zlib_isal": "false", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "compressor_zlib_level": "5", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "compressor_zlib_winsize": "-15", 2024-09-11T05:51:03.918 INFO:teuthology.orchestra.run.smithi165.stdout: "compressor_zstd_level": "1", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_alien_op_num_threads": "6", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_osd_stat_interval": "0", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_seastar_cpu_cores": "", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crimson_seastar_num_threads": "0", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_location": "", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_location_hook": "", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_location_hook_timeout": "10", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "daemonize": "false", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "daos_pool": "tank", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-11T05:51:03.919 INFO:teuthology.orchestra.run.smithi165.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_alienstore": "0/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_allow_any_pool_priority": "false", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_asok": "1/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_asok_assert_abort": "false", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_asserts_on_shutdown": "true", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_auth": "20/20", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_bdev": "1/3", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_bluefs": "1/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_bluestore": "1/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_buffer": "0/1", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_ceph_exporter": "1/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_cephfs_mirror": "0/5", 2024-09-11T05:51:03.920 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_cephsqlite": "0/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_client": "0/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_compressor": "1/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_context": "0/1", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_crimson_interrupt": "0/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_crush": "1/1", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_crypto": "1/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_cyanstore": "0/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_deliberately_leak_memory": "false", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_disable_randomized_ping": "false", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_dpdk": "1/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_eventtrace": "1/5", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_filer": "0/1", 2024-09-11T05:51:03.921 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_filestore": "1/3", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_finisher": "1/1", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_fuse": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_heartbeat_testing_span": "0", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_heartbeatmap": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_javaclient": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_journal": "1/3", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_journaler": "0/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_kstore": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_lockdep": "0/1", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mclock": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_balancer": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_locker": "1/5", 2024-09-11T05:51:03.922 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_log": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_log_expire": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_migrator": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mds_quiesce": "3/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_memstore": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mgr": "2/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mgrc": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_mon": "20/20", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_monc": "0/10", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_ms": "1/1", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_none": "0/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_objclass": "0/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_objectcacher": "0/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_objecter": "0/1", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_optracker": "0/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_osd": "1/5", 2024-09-11T05:51:03.923 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_paxos": "20/20", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_perfcounter": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_prioritycache": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rados": "0/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rbd": "0/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rbd_mirror": "0/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rbd_pwl": "0/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rbd_replay": "0/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_refs": "0/0", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_reserver": "1/1", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_access": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_datacache": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_dbstore": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_flight": "1/5", 2024-09-11T05:51:03.924 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rgw_sync": "1/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_rocksdb": "4/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_backref": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_cache": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_cleaner": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_device": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_epm": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_journal": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_lba": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_odata": "0/5", 2024-09-11T05:51:03.925 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_omap": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_onode": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_t": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_seastore_tm": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_striper": "0/1", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_test": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_throttle": "1/1", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_timer": "0/1", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_tp": "0/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "debug_trace": "1/5", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "device_failure_prediction_mode": "none", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "err_to_graylog": "false", 2024-09-11T05:51:03.926 INFO:teuthology.orchestra.run.smithi165.stdout: "err_to_journald": "false", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "err_to_stderr": "true", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "err_to_syslog": "false", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event_tracing": "false", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_addr": "0.0.0.0", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_http_port": "9926", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_prio_limit": "5", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_sort_metrics": "true", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "exporter_stats_period": "5", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "fake_statfs_for_testing": "0", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "fatal_signal_handlers": "true", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "filer_max_purge_ops": "10", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "filer_max_truncate_ops": "128", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_apply_finisher_threads": "1", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_blackhole": "false", 2024-09-11T05:51:03.927 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_btrfs_clone_range": "true", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_btrfs_snap": "true", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_caller_concurrency": "10", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_collect_device_partition_information": "true", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_commit_timeout": "600.000000", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_debug_inject_read_err": "false", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_debug_omap_check": "false", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_debug_verify_split": "false", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_dump_file": "", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fadvise": "true", 2024-09-11T05:51:03.928 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fail_eio": "true", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fd_cache_shards": "16", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fd_cache_size": "128", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fiemap": "false", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fiemap_threshold": "4096", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_inject_stall": "0", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_journal_parallel": "false", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_journal_trailing": "false", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_journal_writeahead": "false", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_kill_at": "0", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-11T05:51:03.929 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattrs": "0", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_xattr_value_size": "0", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_merge_threshold": "-10", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_odsync_write": "false", 2024-09-11T05:51:03.930 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_omap_backend": "rocksdb", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_omap_backend_path": "", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_op_thread_timeout": "60", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_op_threads": "2", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_punch_hole": "false", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-11T05:51:03.931 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_queue_max_ops": "50", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_seek_data_hole": "false", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_sloppy_crc": "false", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_splice": "false", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_split_multiple": "2", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_split_rand_factor": "20", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_update_to": "1000", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-11T05:51:03.932 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_enable": "true", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_xfs_extsize": "false", 2024-09-11T05:51:03.933 INFO:teuthology.orchestra.run.smithi165.stdout: "filestore_zfs_snap": "false", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fio_dir": "/tmp/fio", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_allow_other": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_atomic_o_trunc": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_big_writes": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_debug": "false", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_default_permissions": "false", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_disable_pagecache": "false", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_max_write": "0", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_multithreaded": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_require_active_mds": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_set_user_groups": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_splice_move": "true", 2024-09-11T05:51:03.934 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_splice_read": "true", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_splice_write": "true", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "fuse_use_invalidate_cb": "true", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "gss_target_name": "ceph", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_file": "", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_inject_failure": "0", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_interval": "5", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "host": "smithi165", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-11T05:51:03.935 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "inject_early_sigterm": "false", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "jaeger_agent_port": "6799", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "jaeger_tracing_enable": "false", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_aio": "true", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_align_min_size": "65536", 2024-09-11T05:51:03.936 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_block_align": "true", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_block_size": "4096", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_dio": "true", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_discard": "false", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_force_aio": "false", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_ignore_corruption": "false", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_max_write_bytes": "10485760", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_max_write_entries": "100", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_replay_from": "0", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_write_header_frequency": "0", 2024-09-11T05:51:03.937 INFO:teuthology.orchestra.run.smithi165.stdout: "journal_zero_on_create": "false", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "journaler_prefetch_periods": "10", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "journaler_prezero_periods": "5", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "journaler_write_head_interval": "15", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "key": "", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "keyfile": "", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_backend": "rocksdb", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_default_stripe_size": "65536", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_fsck_on_mount": "false", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_max_bytes": "67108864", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_max_ops": "512", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_nid_prealloc": "1024", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_onode_map_size": "1024", 2024-09-11T05:51:03.938 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_sync_submit_transaction": "false", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "kstore_sync_transaction": "false", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "librados_thread_count": "2", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "lockdep": "false", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "lockdep_force_backtrace": "false", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_coarse_timestamps": "true", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_flush_on_exit": "false", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_graylog_host": "127.0.0.1", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_graylog_port": "12201", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_max_new": "1000", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_max_recent": "10000", 2024-09-11T05:51:03.939 INFO:teuthology.orchestra.run.smithi165.stdout: "log_stderr_prefix": "", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_stop_at_utilization": "0.970000", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_to_file": "true", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_to_graylog": "false", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_to_journald": "false", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_to_stderr": "false", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "log_to_syslog": "false", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "max_rotating_auth_attempts": "10", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_action_on_write_error": "1", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_alternate_name_max": "8192", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_asio_thread_count": "2", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_export_pin": "true", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_fragment_dirs": "true", 2024-09-11T05:51:03.940 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_fragment_interval": "5", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_interval": "10", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_max": "-1", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_max_until": "-1", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_merge_size": "50", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_midchunk": "0.300000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_min_start": "0.200000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_minchunk": "0.001000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_mode": "0", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_need_max": "1.200000", 2024-09-11T05:51:03.941 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_need_min": "0.800000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_overload_epochs": "2", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_split_bits": "3", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_split_size": "10000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_target_decay": "10.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_beacon_grace": "15.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_beacon_interval": "4.000000", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-11T05:51:03.942 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_mid": "0.700000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_quiesce_delay": "0", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_release_free_interval": "10", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_reservation": "0.050000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_trim_interval": "1", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache_trim_threshold": "262144", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_client_prealloc_inos": "1000", 2024-09-11T05:51:03.943 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_connect_bootstrapping": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_damage_table_max_entries": "10000", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_debug_auth_pins": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_debug_frag": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_debug_scatterstat": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_debug_subtrees": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_decay_halflife": "5.000000", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_default_dir_hash": "2", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_defer_session_stale": "true", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_deny_all_reconnect": "false", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dir_keys_per_op": "16384", 2024-09-11T05:51:03.944 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dir_max_commit_size": "10", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dir_max_entries": "0", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dir_prefetch": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dump_cache_on_map": "false", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_early_reply": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_enable_op_tracker": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_enforce_unique_name": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_export_ephemeral_random": "true", 2024-09-11T05:51:03.945 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_health_summarize_threshold": "10", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_health_dummy": "false", 2024-09-11T05:51:03.946 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_migrator_session_race": "false", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_join_fs": "", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_journal_format": "1", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_create_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_export_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_import_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_journal_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_journal_expire_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_journal_replay_at": "0", 2024-09-11T05:51:03.947 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_link_at": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_mdstable_at": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_openc_at": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_rename_at": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_kill_shutdown_at": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_event_large_threshold": "524288", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_events_per_segment": "1024", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_max_events": "-1", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_max_segments": "128", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_pause": "false", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_segment_size": "0", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-11T05:51:03.948 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_trim_threshold": "128", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log_warn_factor": "2.000000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_caps_per_client": "1048576", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_completed_flushes": "100000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_completed_requests": "100000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_export_size": "20971520", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_file_recover": "32", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_purge_files": "64", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_purge_ops": "8192", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_max_snaps_per_dir": "100", 2024-09-11T05:51:03.949 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_metrics_update_interval": "2", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_min_caps_per_client": "100", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_min_caps_working_set": "10000", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_numa_node": "-1", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_complaint_time": "30.000000", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_history_duration": "600", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_history_size": "20", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_history_slow_op_size": "20", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_op_log_threshold": "5", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_ping_grace": "15", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_ping_interval": "5", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-11T05:51:03.950 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_max_caps": "30000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_recall_warning_threshold": "262144", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_replay_interval": "1.000000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_root_ino_gid": "0", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_root_ino_uid": "0", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-11T05:51:03.951 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_shutdown_check": "0", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_skip_ino": "0", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_snap_max_uid": "4294967294", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_snap_min_uid": "0", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_snap_rstat": "false", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_standby_replay_damaged": "false", 2024-09-11T05:51:03.952 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_symlink_recovery": "true", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_thrash_exports": "0", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_thrash_fragments": "0", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_tick_interval": "5.000000", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_valgrind_exit": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_verify_backtrace": "1", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_verify_scatter": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_wipe_sessions": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool_debug": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "memstore_device_bytes": "1073741824", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "memstore_page_set": "false", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "memstore_page_size": "65536", 2024-09-11T05:51:03.953 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_client_bytes": "134217728", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_client_messages": "512", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_disabled_modules": "", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_enable_op_tracker": "true", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_max_pg_num_change": "128", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_mds_bytes": "134217728", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_mds_messages": "128", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_mon_bytes": "134217728", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_mon_messages": "128", 2024-09-11T05:51:03.954 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_history_duration": "600", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_history_size": "20", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_op_log_threshold": "5", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_osd_bytes": "536870912", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_osd_messages": "8192", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_pool": "true", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_standby_modules": "true", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_stats_period": "5", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_stats_threshold": "5", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_tick_period": "2", 2024-09-11T05:51:03.955 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_allow_pool_delete": "true", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_allow_pool_size_one": "true", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_auth_validate_all_caps": "true", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_bytes": "104857600", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_directed_command_retry": "5", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-11T05:51:03.956 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_hunt_parallel": "3", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_log_interval": "1.000000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_ping_interval": "10.000000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_client_target_rank": "-1", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_file_level": "debug", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_file": "true", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-11T05:51:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_journald": "false", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_compact_on_bootstrap": "false", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_compact_on_start": "false", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_compact_on_trim": "true", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_cpu_threads": "4", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_crush_min_required_version": "hammer", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_daemon_bytes": "419430400", 2024-09-11T05:51:03.958 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_data_avail_crit": "5", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_data_avail_warn": "5", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_data_size_warn": "16106127360", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_dump_json": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_dump_transactions": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_extra_checks": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_no_require_reef": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_no_require_squid": "false", 2024-09-11T05:51:03.959 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_down_mkfs_grace": "120", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_down_uptime_grace": "60", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_election_default_strategy": "1", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_election_timeout": "5.000000", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_elector_ping_divisor": "2", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_enable_op_tracker": "true", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_fake_pool_delete": "false", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_force_quorum_join": "false", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_globalid_prealloc": "10000", 2024-09-11T05:51:03.960 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_detail_to_clog": "false", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_log_update_period": "5", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_max_detail": "50", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_to_clog": "true", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_to_clog_interval": "600", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_host": "172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],172.21.15.172", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_host_override": "", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_initial_members": "", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_lease": "5.000000", 2024-09-11T05:51:03.961 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_log_full_interval": "50", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_log_max": "10000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_log_max_summary": "50", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_log_epochs": "500", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_osd": "10000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_pg_per_osd": "10000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_pool_pg_num": "65536", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-11T05:51:03.962 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mds_force_trim_to": "0", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mds_skip_sanity": "false", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_memory_autotune": "true", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_memory_target": "2147483648", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_beacon_grace": "30", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_digest_period": "5", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_inactive_grace": "60", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_min_osdmap_epochs": "500", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_complaint_time": "30", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_history_duration": "600", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_history_size": "20", 2024-09-11T05:51:03.963 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_history_slow_op_size": "20", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_op_log_threshold": "5", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_auto_mark_in": "false", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_cache_size": "500", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-11T05:51:03.964 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_down_out_interval": "600", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_force_trim_to": "0", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_min_down_reporters": "2", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-11T05:51:03.965 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_report_timeout": "900", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-11T05:51:03.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_stuck_threshold": "60", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_probe_timeout": "2.000000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_reweight_max_change": "0.050000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_reweight_max_osds": "4", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-11T05:51:03.967 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_scrub_interval": "86400", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_scrub_max_keys": "100", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_scrub_timeout": "300", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_session_timeout": "300", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_smart_report_timeout": "5", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_stat_smooth_intervals": "6", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_stretch_pool_min_size": "2", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_stretch_pool_size": "4", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_debug": "false", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-11T05:51:03.968 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_provider_kill_at": "0", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_requester_kill_at": "0", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_sync_timeout": "60.000000", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_target_pg_per_osd": "100", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_tick_interval": "5", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_timecheck_interval": "300.000000", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_use_min_delay_socket": "false", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_older_version_delay": "604800", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-11T05:51:03.969 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_misplaced": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_older_version": "true", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "monmap": "", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T05:51:03.970 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "motr_tracing_enabled": "false", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_op_threads": "3", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_cm": "false", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_device_name": "", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_dscp": "96", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-11T05:51:03.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_local_gid": "", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_port_num": "1", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_sl": "3", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_support_srq": "true", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_rdma_type": "ib", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_async_reap_threshold": "5", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_before_connect": "false", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_ipv4": "true", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_ipv6": "false", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_msgr1": "true", 2024-09-11T05:51:03.972 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_msgr2": "true", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_port_max": "7568", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_port_min": "6800", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_retry_count": "3", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_bind_retry_delay": "5", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_blackhole_client": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_blackhole_mds": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_blackhole_mgr": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_blackhole_mon": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_blackhole_osd": "false", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_client_mode": "crc secure", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_cluster_mode": "crc secure", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_cluster_type": "", 2024-09-11T05:51:03.973 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_compress_secure": "false", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_connection_idle_timeout": "900", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_connection_ready_timeout": "10", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_crc_data": "true", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_crc_header": "true", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_die_on_bad_msg": "false", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_die_on_bug": "true", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_die_on_old_message": "true", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_die_on_skipped_message": "false", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_coremask": "0xF", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-11T05:51:03.974 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_enable_tso": "true", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_hugepages": "", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_lro": "true", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_memory_channel": "4", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_pmd": "", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_port_id": "0", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_dump_on_send": "false", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_initial_backoff": "0.200000", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_delay_max": "1.000000", 2024-09-11T05:51:03.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_delay_type": "", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_network_congestion": "0", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_inject_socket_failures": "5000", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_learn_addr_from_peer": "true", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_max_accept_failures": "4", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_max_backoff": "15.000000", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_mon_client_mode": "secure crc", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_mon_service_mode": "secure crc", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_osd_compress_min_size": "1024", 2024-09-11T05:51:03.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_osd_compress_mode": "none", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_pq_min_cost": "65536", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_public_type": "", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_service_mode": "crc secure", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_tcp_listen_backlog": "512", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_tcp_nodelay": "true", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_tcp_rcvbuf": "0", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ms_type": "async+posix", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "no_config_file": "false", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_completion_locks_per_session": "32", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-11T05:51:03.977 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_inflight_ops": "1024", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_tick_interval": "5.000000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter_timeout": "10.000000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objectstore_blackhole": "false", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "openssl_engine_opts": "", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_delay_time": "5.000000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_hist_halflife": "1000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_max_low_ops": "2", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_max_ops": "4", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_agent_slop": "0.020000", 2024-09-11T05:51:03.978 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_asio_thread_count": "2", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_async_recovery_min_cost": "100", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backfill_scan_max": "512", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backfill_scan_min": "64", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backoff_on_degraded": "false", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backoff_on_peering": "false", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_backoff_on_unfound": "true", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_beacon_report_interval": "300", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bench_duration": "30", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-11T05:51:03.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bench_max_block_size": "67108864", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_blkin_trace_all": "false", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_check_for_log_corruption": "false", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_class_update_on_start": "true", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_client_message_cap": "256", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_client_message_size_cap": "524288000", 2024-09-11T05:51:03.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_client_op_priority": "63", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_client_watch_timeout": "30", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_command_max_records": "256", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_compact_on_start": "false", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_crush_update_on_start": "true", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_crush_update_weight_set": "true", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-11T05:51:03.981 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_feed_pullee": "-1", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_misdirected_ops": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_no_acting_change": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_no_purge_strays": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_op_order": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-11T05:51:03.982 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_shutdown": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_trim_objects": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_verify_snaps": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_keys": "1024", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_stride": "524288", 2024-09-11T05:51:03.983 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_default_notify_timeout": "30", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_delete_sleep": "0.000000", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_discard_disconnected_ops": "true", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_enable_op_tracker": "true", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_fast_info": "true", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_fast_shutdown": "true", 2024-09-11T05:51:03.984 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_function_tracing": "false", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_grace": "20", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_interval": "6", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_min_peers": "10", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_min_size": "2000", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_stale": "600", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_hit_set_max_size": "100000", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_hit_set_min_size": "1000", 2024-09-11T05:51:03.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_journal_size": "5120", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_kill_backfill_at": "0", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_map_cache_size": "50", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_map_dedup": "true", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_map_message_max": "40", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_map_share_max_epochs": "40", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_attr_name_len": "100", 2024-09-11T05:51:03.986 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_attr_size": "0", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_backfills": "1", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_markdown_count": "5", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_markdown_period": "600", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_object_name_len": "2048", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_object_namespace_len": "256", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_object_size": "134217728", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_pg_blocked_by": "16", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_pg_log_entries": "10000", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_pgls": "1024", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_push_cost": "8388608", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_push_objects": "10", 2024-09-11T05:51:03.987 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_scrubs": "3", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_trimming_pgs": "2", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_write_op_reply_len": "64", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_max_write_size": "90", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_profile": "balanced", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-11T05:51:03.988 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_base": "805306368", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_cache_min": "134217728", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-11T05:51:03.989 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_target": "4294967296", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_target_autotune": "false", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_min_pg_log_entries": "250", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_min_recovery_priority": "0", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mon_report_interval": "5", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_num_cache_shards": "32", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_num_op_tracker_shard": "32", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_numa_auto_affinity": "true", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_numa_node": "-1", 2024-09-11T05:51:03.990 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_numa_prefer_iface": "true", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_objecter_finishers": "1", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_objectstore": "bluestore", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_objectstore_fuse": "false", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_objectstore_tracing": "false", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_complaint_time": "30.000000", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_history_duration": "600", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_history_size": "20", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_history_slow_op_size": "20", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_log_threshold": "5", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_shards": "0", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_shards_hdd": "5", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_shards_ssd": "8", 2024-09-11T05:51:03.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_pq_min_cost": "65536", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_queue_cut_off": "high", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_op_thread_timeout": "15", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_open_classes_on_start": "true", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_os_flags": "0", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_peering_op_priority": "255", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_delete_cost": "1048576", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_delete_priority": "5", 2024-09-11T05:51:03.992 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_log_trim_max": "10000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_log_trim_min": "100", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-11T05:51:03.993 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_crimson": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_flags": "0", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_min_size": "0", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-11T05:51:03.994 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_pg_num": "32", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_pgp_num": "0", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_read_ratio": "70", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_size": "2", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_default_type": "replicated", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_push_per_object_cost": "1000", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recover_clone_overlap": "true", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_cost": "20971520", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-11T05:51:03.995 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_active": "0", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_max_single_start": "1", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_op_priority": "3", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_priority": "5", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_sleep": "0.000000", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-11T05:51:03.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_repair_during_recovery": "false", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_requested_scrub_priority": "120", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_auto_repair": "false", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_begin_hour": "0", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_begin_week_day": "0", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_chunk_max": "25", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_chunk_min": "5", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_cost": "52428800", 2024-09-11T05:51:03.997 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_during_recovery": "false", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_end_hour": "0", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_end_week_day": "0", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_event_cost": "4096", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_invalid_stats": "true", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_max_preemptions": "5", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_priority": "5", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_scrub_sleep": "0.000000", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-11T05:51:03.998 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_skip_data_digest": "false", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_smart_report_timeout": "5", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_cost": "1048576", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_priority": "5", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_target_transaction_size": "30", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-11T05:51:03.999 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_tracing": "false", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_use_stale_snap": "false", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "osdc_blkin_trace_all": "false", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_kill_at": "0", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_max_join_drift": "10", 2024-09-11T05:51:04.000 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_min": "500", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_min_wait": "0.050000", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_propose_interval": "1.000000", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_service_trim_max": "500", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_service_trim_min": "250", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_stash_full_interval": "25", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_trim_max": "500", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "paxos_trim_min": "250", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "perf": "true", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "-", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrv": "", 2024-09-11T05:51:04.001 INFO:teuthology.orchestra.run.smithi165.stdout: "public_bind_addr": "-", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "public_network": "", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "public_network_interface": "", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "qat_compressor_busy_polling": "false", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "qat_compressor_enabled": "false", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "qat_compressor_session_max_number": "256", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rados_mon_op_timeout": "0", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rados_osd_op_timeout": "0", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rados_tracing": "false", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_atime_update_interval": "60", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_balance_parent_reads": "false", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_balance_snap_reads": "false", 2024-09-11T05:51:04.002 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_blkin_trace_all": "false", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache": "true", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_policy": "writearound", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_size": "33554432", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_clone_copy_on_read": "false", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_compression_hint": "none", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_concurrent_management_ops": "10", 2024-09-11T05:51:04.003 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_clone_format": "auto", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_data_pool": "", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_features": "61", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_format": "2", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_map_options": "", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_order": "22", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_pool": "rbd", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_stripe_count": "0", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_default_stripe_unit": "0", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-11T05:51:04.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_enable_alloc_hint": "true", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_io_scheduler": "simple", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_order": "24", 2024-09-11T05:51:04.005 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_pool": "", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_journal_splay_width": "4", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_localize_parent_reads": "false", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_localize_snap_reads": "false", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-11T05:51:04.006 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-11T05:51:04.007 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_mtime_update_interval": "60", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_non_blocking_aio": "true", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_op_thread_timeout": "60", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_op_threads": "1", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_parent_cache_enabled": "false", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-11T05:51:04.008 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_plugins": "", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_bps_burst": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_bps_limit": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_exclude_ops": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_iops_burst": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_iops_limit": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-11T05:51:04.009 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_read_from_replica_policy": "default", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_skip_partial_discard": "true", 2024-09-11T05:51:04.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_tracing": "false", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_validate_names": "true", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd_validate_pool": "true", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "restapi_base_url": "", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "restapi_log_level": "", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_acl_grants_max_num": "100", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_admin_entry": "admin", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_backend_store": "rados", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_barbican_url": "", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_beast_enable_async": "true", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_counters_cache": "false", 2024-09-11T05:51:04.011 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_cache_enabled": "true", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_cache_expiry_interval": "900", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_cache_lru_size": "25000", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_config_store": "rados", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_content_length_compat": "false", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_copy_obj_progress": "true", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-11T05:51:04.012 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_cors_rules_max_num": "100", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_cross_domain_policy": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_addr": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_password": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_kmip_username": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_require_ssl": "true", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-11T05:51:04.013 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_addr": "", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_auth": "token", 2024-09-11T05:51:04.014 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_namespace": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_prefix": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_token_file": "", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_curl_buffersize": "524288", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_curl_low_speed_time": "300", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-11T05:51:04.015 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_d4n_port": "6379", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_log_changes_size": "1000", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_log_num_shards": "128", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_log_window": "30", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-11T05:51:04.016 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-11T05:51:04.017 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dns_name": "", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dns_s3website_name": "", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_dynamic_resharding": "true", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-11T05:51:04.018 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_gc_threads": "true", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_lc_threads": "true", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_ops_log": "false", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_quota_threads": "true", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_static_website": "false", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enable_usage_log": "false", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_enforce_swift_acls": "true", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_exit_timeout_secs": "120", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_expose_bucket": "false", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_extended_http_attrs": "", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_filter": "none", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_frontends": "beast port=7480", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_deferred": "50", 2024-09-11T05:51:04.019 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_objs": "32", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_gc_processor_period": "3600", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_init_timeout": "300", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_kafka_connection_idle": "300", 2024-09-11T05:51:04.020 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_domain": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_password": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_password_path": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_project": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_tenant": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_token": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_token_path": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_admin_user": "", 2024-09-11T05:51:04.021 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_api_version": "2", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_barbican_domain": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_barbican_password": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_barbican_project": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_barbican_user": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_url": "", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_debug_interval": "-1", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_lock_max_time": "90", 2024-09-11T05:51:04.022 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_max_objs": "32", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_max_rules": "1000", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_max_worker": "3", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lc_thread_delay": "0", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_dnattr": "uid", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_searchfilter": "", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ldap_uri": "", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_log_http_headers": "", 2024-09-11T05:51:04.023 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_log_object_name_utc": "false", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_attr_name_len": "0", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_attr_size": "0", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_chunk_size": "4194304", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_listing_results": "1000", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_notify_retries": "10", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-11T05:51:04.024 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_put_param_size": "1048576", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_put_size": "5368709120", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_max_slo_entries": "1000", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_md_log_max_shards": "64", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_mp_lock_max_time": "600", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-11T05:51:04.025 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_max_gc": "300", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_num_async_rados_threads": "32", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_num_control_oids": "8", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_numa_node": "-1", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-11T05:51:04.026 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_objexp_chunk_size": "100", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_objexp_gc_interval": "600", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_op_thread_timeout": "600", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_op_tracing": "false", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_opa_token": "", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_opa_url": "", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_opa_verify_ssl": "true", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ops_log_rados": "false", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_ops_log_socket_path": "", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-11T05:51:04.027 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_period_push_interval": "2.000000", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_cache_lanes": "3", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_cache_partitions": "3", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_print_continue": "true", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-11T05:51:04.028 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_rados_tracing": "false", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_realm": "", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_realm_id": "", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_region": "", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_request_uri": "", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_reshard_batch_size": "64", 2024-09-11T05:51:04.029 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_reshard_max_aio": "128", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_reshard_num_logs": "16", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_reshard_thread_interval": "600", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_resolve_cname": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_run_sync_thread": "true", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-11T05:51:04.030 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_scheduler_type": "throttler", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_script_uri": "", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_service_provider_name": "", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_client_id": "", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_client_secret": "", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_entry": "sts", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_key": "sts", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_min_session_duration": "900", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sts_token_introspection_url": "", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_account_in_url": "false", 2024-09-11T05:51:04.031 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_auth_entry": "auth", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_auth_url": "", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_custom_header": "", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_need_stats": "true", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_tenant_name": "", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_token_expiration": "86400", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_url": "", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_url_prefix": "swift", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-11T05:51:04.032 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_lease_period": "120", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_thread_pool_size": "512", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-11T05:51:04.033 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_comment": "", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_createby": "", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_encoding": "", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_flag": "false", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_origin": "", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_torrent_tracker": "", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_trust_forwarded_https": "false", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_usage_max_shards": "32", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_usage_max_user_shards": "1", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_use_opa_authz": "false", 2024-09-11T05:51:04.034 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_counters_cache": "false", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_max_buckets": "1000", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_policies_max_num": "100", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_user_unique_email": "true", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_verify_ssl": "true", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-11T05:51:04.035 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zone": "", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zone_id": "", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zonegroup": "", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zonegroup_id": "", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgwlc_auto_session_clear": "true", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_block_size": "4096", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-11T05:51:04.036 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_size": "536870912", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_index_type": "binary_search", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_partition_filters": "false", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_perf": "false", 2024-09-11T05:51:04.037 INFO:teuthology.orchestra.run.smithi165.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "run_dir": "/var/run/ceph", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_block_create": "true", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_cache_lru_size": "67108864", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_default_max_object_size": "16777216", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_device_size": "53687091200", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_full_integrity_check": "false", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_journal_batch_capacity": "16", 2024-09-11T05:51:04.038 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_main_device_type": "SSD", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "seastore_segment_size": "67108864", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "set_keepcaps": "false", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "setgroup": "", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "setuser": "", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "setuser_match_path": "", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-11T05:51:04.039 INFO:teuthology.orchestra.run.smithi165.stdout: "thp": "false", 2024-09-11T05:51:04.040 INFO:teuthology.orchestra.run.smithi165.stdout: "threadpool_default_timeout": "60", 2024-09-11T05:51:04.040 INFO:teuthology.orchestra.run.smithi165.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-11T05:51:04.040 INFO:teuthology.orchestra.run.smithi165.stdout: "throttler_perf_counter": "true" 2024-09-11T05:51:04.040 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:04.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-11T05:51:04.177 INFO:teuthology.orchestra.run.smithi165.stdout:mon.a: {} 2024-09-11T05:51:04.177 INFO:teuthology.orchestra.run.smithi165.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:51:04.189 INFO:teuthology.orchestra.run.smithi165.stdout:mon.b: {} 2024-09-11T05:51:04.189 INFO:teuthology.orchestra.run.smithi165.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:51:04.201 INFO:teuthology.orchestra.run.smithi165.stdout:mon.c: {} 2024-09-11T05:51:04.201 INFO:teuthology.orchestra.run.smithi165.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:51:04.215 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-09-11T05:51:04.222 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T05:51:04.222 DEBUG:teuthology.orchestra.run.smithi165:> type ceph-objectstore-tool 2024-09-11T05:51:04.225 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T05:51:04.226 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-09-11T05:51:04.235 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T05:51:04.236 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T05:51:04.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-11T05:51:04.237 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-11T05:51:04.237 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-11T05:51:04.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:04.237 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-11T05:51:04.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:04.238 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-11T05:51:04.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:51:04.239 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-09-11T05:51:04.265 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:04.266+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:04.285 INFO:teuthology.orchestra.run.smithi165.stdout:mapped writes DISABLED 2024-09-11T05:51:04.285 INFO:teuthology.orchestra.run.smithi165.stdout:mapped reads DISABLED 2024-09-11T05:51:04.285 INFO:teuthology.orchestra.run.smithi165.stdout:Seed set to 7926 2024-09-11T05:51:04.340 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:04.338+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:04.382 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:04.383 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:04.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:04.440 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:04.442+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:04.455 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:04.455 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:04.487 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:04.487 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:04.505 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:04.505 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:04.521 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:04.521 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:04.526 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:04.526 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:04.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:04.540 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:04.538+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:04.541 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:04.542 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:04.562 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:04.562 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:04.582 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:04.583 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:04.603 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:04.604 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:04.647 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:04.647 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:04.648 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:04.646+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:04.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:04.751 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:04.750+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:04.812 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:04.812 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:04.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:04.867 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:04.866+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:04.945 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:04.942+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:04.972 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:04.973 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:04.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:05.046 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:05.046+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:05.142 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.143 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.146 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:05.146+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:05.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:05.206 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:51:05.247 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:05.246+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:05.278 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.278 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:05.348 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:05.346+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:05.436 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.436 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:05.449 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:05.446+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:05.549 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:05.550+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:05.609 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.610 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:05.650 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:05.650+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:05.740 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.740 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.751 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:05.750+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:05.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:05.851 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:05.850+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:05.870 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:05.870 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:05.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:05.952 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:05.954+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:06.042 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.042 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.053 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:06.054+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:06.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:06.154 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:06.154+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:06.186 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.186 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:06.255 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:06.254+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:06.319 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.319 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:06.381 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:06.382+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:06.451 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.451 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.457 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:06.458+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:06.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:06.557 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:06.554+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:06.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:06.633 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.633 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:06.667 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:06.666+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:06.759 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:06.758+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:06.772 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:06.772 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:06.774 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:06.775 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:06.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:06.791 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:06.791 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:06.807 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:06.808 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:06.824 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:06.824 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:06.843 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:06.843 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:06.859 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:06.858+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:06.861 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:06.861 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:06.879 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:06.879 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:06.897 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:06.897 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4239.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:56.149555+0000", 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.80449237, 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012799534, 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4239", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1919891914", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149555+0000", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149555+0000", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149561+0000", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2230000000000004e-06 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149575+0000", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.431e-05 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149606+0000", 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0505999999999999e-05 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149632+0000", 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.582e-05 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149840+0000", 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208035 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.149875+0000", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.506e-05 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.153869+0000", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039947130000000004 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.153892+0000", 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.158492+0000", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046000900000000003 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.158736+0000", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024446199999999998 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.159092+0000", 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035546699999999999 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161449+0000", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023573380000000001 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.162150+0000", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070072199999999998 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.162277+0000", 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012682200000000001 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.162354+0000", 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7688999999999994e-05 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(client.4239.0:1 2.3 e20/19 ondisk, result = 0)", 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:56.161268+0000", 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.792778569999999, 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019303500000000001, 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.960 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161268+0000", 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161268+0000", 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161270+0000", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2929999999999999e-06 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161278+0000", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5900000000000008e-06 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161300+0000", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2036e-05 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161314+0000", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3740000000000001e-05 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161426+0000", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111802 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161440+0000", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4246999999999999e-05 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.161461+0000", 2024-09-11T05:51:06.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1327000000000001e-05 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4239.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20)", 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:56.167091+0000", 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.786955679, 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000396587, 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4239", 2024-09-11T05:51:06.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1919891914", 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167091+0000", 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167091+0000", 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167094+0000", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.683e-06 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167106+0000", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1897e-05 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167138+0000", 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1792999999999999e-05 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167153+0000", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5227e-05 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167273+0000", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119639 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.167488+0000", 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215348 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.967 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4239.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20)", 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:56.168068+0000", 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.785979143, 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040354699999999998, 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4239", 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1919891914", 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3 2024-09-11T05:51:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168068+0000", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168068+0000", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168070+0000", 2024-09-11T05:51:06.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2069999999999998e-06 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168081+0000", 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1226000000000001e-05 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168113+0000", 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.137e-05 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168138+0000", 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5133000000000001e-05 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168279+0000", 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141049 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:56.168471+0000", 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019256199999999999 2024-09-11T05:51:06.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4239.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:57.162279+0000", 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.791767536, 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072035240000000002, 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4239", 2024-09-11T05:51:06.972 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1919891914", 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 5 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162279+0000", 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162279+0000", 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162283+0000", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2930000000000001e-06 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162301+0000", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8634000000000002e-05 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162336+0000", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4462999999999999e-05 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162353+0000", 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6903000000000001e-05 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162501+0000", 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147849 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.162512+0000", 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1643e-05 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.166258+0000", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037453080000000001 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.166510+0000", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252878 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.166716+0000", 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020560500000000001 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.168668+0000", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019518420000000001 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169334+0000", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00066642399999999999 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169418+0000", 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3992999999999994e-05 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169483+0000", 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4689000000000004e-05 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(client.4239.0:5 2.2 e21/19 ondisk, result = 0)", 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:57.169148+0000", 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7848994259999991, 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035007400000000002, 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:06.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169148+0000", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169148+0000", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169150+0000", 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9999999999999999e-06 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169160+0000", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0107e-05 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169189+0000", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9482999999999999e-05 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169204+0000", 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4436999999999999e-05 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169313+0000", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109209 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169326+0000", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2972999999999999e-05 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:57.169498+0000", 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171865 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.981 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4239.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:58.169255+0000", 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.7847918830000005, 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073906989999999997, 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4239", 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1919891914", 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 7 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169255+0000", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169255+0000", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169258+0000", 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0759999999999999e-06 2024-09-11T05:51:06.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169274+0000", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6160000000000001e-05 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169310+0000", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5283000000000001e-05 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169327+0000", 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7269999999999999e-05 2024-09-11T05:51:06.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169464+0000", 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013726299999999999 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.169475+0000", 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0626e-05 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.173465+0000", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039903489999999998 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.173616+0000", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015042899999999999 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.173862+0000", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024680500000000001 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:06.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.175950+0000", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020878540000000001 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176475+0000", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052452200000000003 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176569+0000", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3852999999999997e-05 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176646+0000", 2024-09-11T05:51:06.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7208999999999999e-05 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(client.4239.0:7 2.2 e21/19 ondisk, result = 0)", 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:58.176289+0000", 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.7777575940000006, 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037044699999999999, 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176289+0000", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176289+0000", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176292+0000", 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999998e-06 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176303+0000", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1463e-05 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176340+0000", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6832999999999998e-05 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176360+0000", 2024-09-11T05:51:06.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0066999999999999e-05 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176450+0000", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0241999999999994e-05 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176464+0000", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3903999999999999e-05 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:58.176660+0000", 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019583799999999999 2024-09-11T05:51:06.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.992 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:06.992 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:06.992 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:06.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:06.993 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:06.990+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:07.093 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:07.094+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:07.112 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.113 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:07.113 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:07.113 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:07.194 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:07.194+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:07.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:51:07.244 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:07.244 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:07.244 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:07.244 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:07.295 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:07.294+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:07.381 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [] 2024-09-11T05:51:07.381 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:07.397 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:07.394+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:07.497 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:07.494+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:07.570 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:07.570 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:07.598 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:07.594+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:07.698 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:07.694+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:07.708 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.708 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:07.708 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:07.708 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:07.708 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:07.800 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:07.798+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.669473982, 2024-09-11T05:51:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011567851000000001, 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179401+0000", 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.573e-06 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179413+0000", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2670000000000001e-05 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179469+0000", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5903000000000003e-05 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179485+0000", 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.186952+0000", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074664290000000001 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187123+0000", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017143900000000001 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187336+0000", 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000212692 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190287+0000", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029509169999999999 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190763+0000", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000475799 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190894+0000", 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013149599999999999 2024-09-11T05:51:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190967+0000", 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2682999999999995e-05 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.669358320000001, 2024-09-11T05:51:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011710134000000001, 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179516+0000", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179529+0000", 2024-09-11T05:51:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2204e-05 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179555+0000", 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6602999999999999e-05 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179576+0000", 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0636999999999999e-05 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187937+0000", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083613920000000005 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188065+0000", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127532 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188273+0000", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020859200000000001 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190983+0000", 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027099189999999999 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191042+0000", 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9008999999999997e-05 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191159+0000", 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116556 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191225+0000", 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6060000000000001e-05 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.573661121000001, 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011718055999999999, 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 87 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275213+0000", 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7300000000000005e-07 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275308+0000", 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5353000000000007e-05 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275320+0000", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1746000000000001e-05 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275328+0000", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2770000000000005e-06 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.282992+0000", 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007663998 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283142+0000", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015011899999999999 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283361+0000", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000218488 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286081+0000", 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027206890000000001 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286782+0000", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070088400000000003 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286863+0000", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1225999999999998e-05 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286930+0000", 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6502999999999997e-05 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.573526046, 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013655390999999999, 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 88 2024-09-11T05:51:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275348+0000", 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275354+0000", 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.947e-06 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275367+0000", 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952999999999999e-05 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275375+0000", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2230000000000007e-06 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284632+0000", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092572620000000005 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284856+0000", 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022421799999999999 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.285198+0000", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034152800000000001 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.287353+0000", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021548800000000001 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288869+0000", 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001515728 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288944+0000", 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5339999999999994e-05 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.289002+0000", 2024-09-11T05:51:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8471999999999999e-05 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.412274767, 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013083005999999999, 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436600+0000", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.553e-06 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436737+0000", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436759+0000", 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1946e-05 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436775+0000", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5767e-05 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444423+0000", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007648874 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444555+0000", 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013180900000000001 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444751+0000", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019567200000000001 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.446939+0000", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021876830000000002 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449422+0000", 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002483791 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449562+0000", 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013971900000000001 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449681+0000", 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119126 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.412067328999999, 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014821173, 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436807+0000", 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.637e-06 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436819+0000", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1663e-05 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436866+0000", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6879999999999998e-05 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436885+0000", 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9483e-05 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447203+0000", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010317244 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447333+0000", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013070200000000001 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447532+0000", 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019884199999999999 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451135+0000", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036030820000000001 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451358+0000", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022260800000000001 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451488+0000", 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130273 2024-09-11T05:51:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451627+0000", 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013875899999999999 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411941453000001, 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017261187000000001, 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436933+0000", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436945+0000", 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1939999999999999e-05 2024-09-11T05:51:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436966+0000", 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0979999999999999e-05 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436981+0000", 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5353e-05 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448139+0000", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011157373999999999 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448357+0000", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021852500000000001 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448717+0000", 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451314+0000", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025974359999999998 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453934+0000", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026193599999999998 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454076+0000", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014191599999999999 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454193+0000", 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117199 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411863427, 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017801221999999998, 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437011+0000", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3999999999999999e-06 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437151+0000", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140359 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437177+0000", 2024-09-11T05:51:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437204+0000", 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7645999999999999e-05 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449807+0000", 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012602197000000001 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449944+0000", 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013775500000000001 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.450149+0000", 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020427499999999999 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454334+0000", 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041850480000000002 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454533+0000", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000198968 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454697+0000", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164446 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454811+0000", 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113775 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411637229, 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017714963, 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437237+0000", 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5570000000000001e-06 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437249+0000", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1800000000000001e-05 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437278+0000", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8866000000000001e-05 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437295+0000", 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7447000000000001e-05 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451717+0000", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014421237999999999 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451859+0000", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014248900000000001 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452057+0000", 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019812200000000001 2024-09-11T05:51:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454491+0000", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024333940000000002 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454830+0000", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033955500000000001 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454900+0000", 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0049000000000006e-05 2024-09-11T05:51:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454951+0000", 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0445999999999998e-05 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411551622999999, 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019404264000000001, 2024-09-11T05:51:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437323+0000", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437333+0000", 2024-09-11T05:51:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0499999999999999e-05 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437356+0000", 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2293000000000002e-05 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437369+0000", 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.381e-05 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452677+0000", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015307215000000001 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452924+0000", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024779199999999999 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453257+0000", 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000332944 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456421+0000", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003164072 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456460+0000", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8869999999999999e-05 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456617+0000", 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156305 2024-09-11T05:51:07.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456726+0000", 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108993 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411432640999999, 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021955514999999998, 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.909 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437441+0000", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8999999999999995e-07 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437529+0000", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8226999999999994e-05 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437556+0000", 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6806000000000001e-05 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437570+0000", 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4003e-05 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455042+0000", 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017471738000000001 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455198+0000", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455397+0000", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019827199999999999 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459080+0000", 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036837860000000001 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459121+0000", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0589000000000001e-05 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459260+0000", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013900199999999999 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459396+0000", 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135933 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411286902000001, 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022184125999999998, 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-09-11T05:51:07.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437587+0000", 2024-09-11T05:51:07.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3699999999999999e-07 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437593+0000", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3829999999999997e-06 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437606+0000", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2649999999999999e-05 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437613+0000", 2024-09-11T05:51:07.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2570000000000001e-06 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456824+0000", 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019210188999999999 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456953+0000", 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012939200000000001 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457153+0000", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000199891 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459412+0000", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022590000000000002 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459524+0000", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111716 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459657+0000", 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013367199999999999 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459770+0000", 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113039 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:07.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.411247802, 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022423287, 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:07.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437626+0000", 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2399999999999997e-07 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437632+0000", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0599999999999996e-06 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437642+0000", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5699999999999999e-06 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437650+0000", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4689999999999997e-06 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457786+0000", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020135574999999999 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458013+0000", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022734200000000001 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458415+0000", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040240300000000002 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459798+0000", 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013821759999999999 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459914+0000", 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116773 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459992+0000", 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8059000000000006e-05 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.460048+0000", 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6035999999999997e-05 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.933414065000001, 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013428412000000001, 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915460+0000", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4759999999999999e-06 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915673+0000", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021259499999999999 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915717+0000", 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4206999999999999e-05 2024-09-11T05:51:07.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915735+0000", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8073000000000001e-05 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.921996+0000", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062606429999999998 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922091+0000", 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4765000000000005e-05 2024-09-11T05:51:07.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922203+0000", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112596 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926133+0000", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003930022 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928670+0000", 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025363069999999998 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928761+0000", 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1853000000000003e-05 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928887+0000", 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012587500000000001 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.933102798, 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013236257, 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915772+0000", 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6559999999999999e-06 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915784+0000", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2034e-05 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915809+0000", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4839999999999999e-05 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915829+0000", 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9809999999999998e-05 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923686+0000", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078569520000000004 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923818+0000", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132182 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924042+0000", 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022383799999999999 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926144+0000", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002102348 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928947+0000", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028030070000000002 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928981+0000", 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4014999999999999e-05 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929006+0000", 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5575000000000002e-05 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.933016619, 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013436350999999999, 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.937 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915858+0000", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5e-06 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915869+0000", 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1039999999999999e-05 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915890+0000", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1056999999999998e-05 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915907+0000", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6898999999999999e-05 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924934+0000", 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090266600000000006 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925029+0000", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5313e-05 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925151+0000", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122416 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926154+0000", 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010029559999999999 2024-09-11T05:51:07.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929245+0000", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003090939 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929272+0000", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6843999999999999e-05 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929293+0000", 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0727e-05 2024-09-11T05:51:07.942 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.932895166, 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016090376999999999, 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.943 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915979+0000", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.623e-06 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916191+0000", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000211385 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916218+0000", 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7503000000000001e-05 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916242+0000", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3657000000000001e-05 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926301+0000", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010058958999999999 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926430+0000", 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129472 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926644+0000", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000213332 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929041+0000", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023971750000000001 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932021+0000", 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029796509999999998 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932048+0000", 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240999999999999e-05 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932068+0000", 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0378999999999999e-05 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.932585379000001, 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016382369000000001, 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916289+0000", 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.747e-06 2024-09-11T05:51:07.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916301+0000", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.201e-05 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916325+0000", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3323e-05 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916364+0000", 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9069999999999997e-05 2024-09-11T05:51:07.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.927959+0000", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011595032 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928047+0000", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7719999999999994e-05 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928196+0000", 2024-09-11T05:51:07.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149276 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929059+0000", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086328399999999997 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932625+0000", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035663819999999999 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932651+0000", 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5732e-05 2024-09-11T05:51:07.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932670+0000", 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8793000000000001e-05 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.19868606, 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011210540999999999, 2024-09-11T05:51:07.954 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 158 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650188+0000", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4300000000000001e-06 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650427+0000", 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023869200000000001 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650453+0000", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6026000000000001e-05 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650475+0000", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.145e-05 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657215+0000", 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067406150000000001 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657330+0000", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114809 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657573+0000", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243241 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660296+0000", 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027227359999999999 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661256+0000", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00095981899999999999 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661329+0000", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2819999999999997e-05 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661397+0000", 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8903000000000001e-05 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.198366059, 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013818587, 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:07.960 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 159 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650509+0000", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.773e-06 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650520+0000", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1202999999999999e-05 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650544+0000", 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4023999999999999e-05 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650576+0000", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1922999999999999e-05 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659257+0000", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086811560000000006 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659395+0000", 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013841299999999999 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659593+0000", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197748 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660722+0000", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011288680000000001 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664169+0000", 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034466929999999998 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664260+0000", 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0946000000000006e-05 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664326+0000", 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5840000000000007e-05 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:07.965 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:07.967 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:07.966+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:07.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:08.067 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:08.066+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:08.085 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.085 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:08.086 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:08.086 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:08.169 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:08.170+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:08.224 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.224 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:08.224 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:08.224 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:08.224 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:08.283 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:08.282+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.162881232, 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003261576, 2024-09-11T05:51:08.350 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185181+0000", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5200000000000001e-06 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185282+0000", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101363 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185303+0000", 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0193000000000001e-05 2024-09-11T05:51:08.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185314+0000", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0923e-05 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185425+0000", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111203 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185454+0000", 2024-09-11T05:51:08.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8955999999999999e-05 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.188334+0000", 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028801719999999999 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.188441+0000", 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107246 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:54 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'32, mlcod=17'29)", 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.126791003000001, 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002724232, 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221272+0000", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221289+0000", 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7410000000000001e-05 2024-09-11T05:51:08.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221325+0000", 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5583999999999997e-05 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221344+0000", 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9599e-05 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221449+0000", 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104393 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221494+0000", 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5636000000000001e-05 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.223907+0000", 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024125209999999999 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.223994+0000", 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7072999999999998e-05 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:68 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'42, mlcod=17'39)", 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.247530+0000", 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.100530526, 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002544064, 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.359 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247530+0000", 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247530+0000", 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247532+0000", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8029999999999999e-06 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247545+0000", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2819999999999999e-05 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247574+0000", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8742999999999999e-05 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247592+0000", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8374000000000001e-05 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247720+0000", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012825900000000001 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.247781+0000", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1036000000000003e-05 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.249983+0000", 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002201433 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.250074+0000", 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1595999999999995e-05 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:83 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'53, mlcod=17'50)", 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:08.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.072149315000001, 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028129919999999998, 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275913+0000", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.147e-06 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275922+0000", 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.713e-06 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275940+0000", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7989999999999999e-05 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275951+0000", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.276061+0000", 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109722 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.276137+0000", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5887e-05 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.278635+0000", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024986259999999999 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.278724+0000", 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9086999999999996e-05 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'54)", 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.065268735, 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027993319999999999, 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:08.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282793+0000", 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5300000000000002e-07 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282854+0000", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1416999999999998e-05 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282875+0000", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0387e-05 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282885+0000", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0373e-05 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282988+0000", 2024-09-11T05:51:08.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102539 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.283041+0000", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2958999999999999e-05 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.285504+0000", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024628279999999998 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.285591+0000", 2024-09-11T05:51:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7875999999999997e-05 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:129 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'87, mlcod=17'85)", 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.434658975, 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028747239999999999, 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913404+0000", 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1859999999999999e-06 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913421+0000", 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6917000000000001e-05 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913459+0000", 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.773e-05 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913478+0000", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9276e-05 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913625+0000", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014670899999999999 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913681+0000", 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5983000000000003e-05 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916208+0000", 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025273230000000002 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916277+0000", 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.86e-05 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:130 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'88, mlcod=17'85)", 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.913847+0000", 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.434213472, 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024754239999999999, 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913847+0000", 2024-09-11T05:51:08.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913847+0000", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913849+0000", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7540000000000001e-06 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913863+0000", 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3859000000000001e-05 2024-09-11T05:51:08.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913889+0000", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6109999999999999e-05 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913907+0000", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7462999999999999e-05 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914487+0000", 2024-09-11T05:51:08.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058069299999999996 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914532+0000", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5179000000000002e-05 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916282+0000", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001749947 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916323+0000", 2024-09-11T05:51:08.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0419e-05 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:131 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'89, mlcod=17'85)", 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.433070681, 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028937379999999999, 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914992+0000", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3799999999999999e-06 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915148+0000", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156859 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915172+0000", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.404e-05 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915185+0000", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2086000000000001e-05 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915261+0000", 2024-09-11T05:51:08.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6342999999999997e-05 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915290+0000", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9390000000000002e-05 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917784+0000", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024937869999999999 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917884+0000", 2024-09-11T05:51:08.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9852999999999994e-05 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:132 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'90, mlcod=17'85)", 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.431305720999999, 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053327920000000003, 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916757+0000", 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4100000000000001e-06 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916767+0000", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0496000000000001e-05 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916790+0000", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.315e-05 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916803+0000", 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2860000000000001e-05 2024-09-11T05:51:08.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916934+0000", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013144900000000001 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917013+0000", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8076000000000003e-05 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922020+0000", 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050075650000000003 2024-09-11T05:51:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922088+0000", 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7786000000000005e-05 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:133 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'91, mlcod=17'85)", 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.430057417, 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058524220000000004, 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.389 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918005+0000", 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2899999999999999e-06 2024-09-11T05:51:08.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918016+0000", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1636e-05 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918045+0000", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8863000000000001e-05 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918058+0000", 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.322e-05 2024-09-11T05:51:08.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918155+0000", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6428999999999994e-05 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918185+0000", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0219999999999999e-05 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923763+0000", 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055783969999999997 2024-09-11T05:51:08.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923856+0000", 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2367000000000005e-05 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:134 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'92, mlcod=17'87)", 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.429128638, 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050191519999999998, 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.393 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918934+0000", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9470000000000002e-06 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919003+0000", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9258999999999995e-05 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919033+0000", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9884000000000001e-05 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919050+0000", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6503000000000001e-05 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919179+0000", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012948199999999999 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919236+0000", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6348999999999999e-05 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923866+0000", 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046299920000000003 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923951+0000", 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5735999999999994e-05 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:135 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'93, mlcod=17'87)", 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.428614404999999, 2024-09-11T05:51:08.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046106410000000004, 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919449+0000", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2170000000000001e-06 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919468+0000", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9490000000000001e-05 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919500+0000", 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1469e-05 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919514+0000", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.486e-05 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920637+0000", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011221149999999999 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920690+0000", 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3328999999999997e-05 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923958+0000", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032681020000000002 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924057+0000", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9059000000000001e-05 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:136 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'94, mlcod=17'87)", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:08.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.427700862, 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037872240000000001, 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920362+0000", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.844e-06 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920374+0000", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2636000000000001e-05 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920403+0000", 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9006e-05 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920417+0000", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.331e-05 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921463+0000", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010457719999999999 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921495+0000", 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2642999999999999e-05 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924065+0000", 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025699239999999999 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924147+0000", 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2089000000000006e-05 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:137 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'95, mlcod=17'88)", 2024-09-11T05:51:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.426480411, 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044436249999999997, 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921582+0000", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.883e-06 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921642+0000", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9735999999999999e-05 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921675+0000", 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2846999999999999e-05 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921688+0000", 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3390000000000001e-05 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922264+0000", 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057572799999999996 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922341+0000", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6779999999999993e-05 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925964+0000", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003623458 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926024+0000", 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9803000000000003e-05 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:138 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'96, mlcod=17'88)", 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.424750251000001, 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027531919999999998, 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923312+0000", 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5030000000000001e-06 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923324+0000", 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1667e-05 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923347+0000", 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3743e-05 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923360+0000", 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3013000000000001e-05 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924276+0000", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091579300000000003 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924332+0000", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6073000000000003e-05 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926027+0000", 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00169487 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926064+0000", 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6529999999999998e-05 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:139 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'97, mlcod=17'89)", 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.423676646000001, 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038371870000000001, 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924386+0000", 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.683e-06 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924400+0000", 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4535999999999999e-05 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924426+0000", 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.533e-05 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924439+0000", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.343e-05 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925265+0000", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082617699999999999 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925354+0000", 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8802999999999997e-05 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928126+0000", 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027723209999999999 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928221+0000", 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4907000000000005e-05 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:140 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'98, mlcod=17'93)", 2024-09-11T05:51:08.418 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.422070732, 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049559219999999998, 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925992+0000", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.533e-06 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926050+0000", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8875999999999997e-05 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926070+0000", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9259999999999999e-05 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926082+0000", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1960000000000001e-05 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926163+0000", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1493000000000001e-05 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926192+0000", 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8583000000000001e-05 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.930860+0000", 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046681140000000001 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.930946+0000", 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6102999999999999e-05 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:141 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'99, mlcod=17'93)", 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.420643093000001, 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041853410000000004, 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927419+0000", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927427+0000", 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-11T05:51:08.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927446+0000", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8683999999999999e-05 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927455+0000", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928283+0000", 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082788000000000002 2024-09-11T05:51:08.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928341+0000", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8183000000000002e-05 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.931544+0000", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032032250000000001 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.931603+0000", 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8990000000000003e-05 2024-09-11T05:51:08.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.82443388399999995, 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033108579999999999, 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:08.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523630+0000", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5969999999999999e-06 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523642+0000", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802999999999999e-05 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523670+0000", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.764e-05 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523684+0000", 2024-09-11T05:51:08.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.431e-05 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523789+0000", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010516100000000001 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.524003+0000", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021329000000000001 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.524306+0000", 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030360000000000001 2024-09-11T05:51:08.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.525515+0000", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012085799999999999 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526790+0000", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001275261 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526874+0000", 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3867000000000002e-05 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526938+0000", 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3749000000000002e-05 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:13 3.12 3:48b35b7a:::rbd_object_map.11246a69a978:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.818702017, 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029318399999999998, 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.432 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 13 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:08.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529361+0000", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9460000000000001e-06 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529372+0000", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1623999999999999e-05 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529399+0000", 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6352000000000001e-05 2024-09-11T05:51:08.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529412+0000", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3647e-05 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529518+0000", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105463 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529744+0000", 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022606500000000001 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529997+0000", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025276399999999999 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.531513+0000", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015163150000000001 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532144+0000", 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063138500000000004 2024-09-11T05:51:08.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532227+0000", 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3035999999999996e-05 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532291+0000", 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3243000000000004e-05 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.437 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:08.439 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:08.438+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:08.500 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:51:08.540 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:08.538+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:08.605 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.605 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:08.605 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:08.605 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:08.640 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:08.642+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:08.741 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:08.742+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:08.744 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:08.745 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:08.745 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:08.842 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:08.842+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:08.865 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:5 2.2 e21/19 2:5cea7035:::rbd_info:head v 21'1, mlcod=0'0)", 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.700774388999999, 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017785089999999999, 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166260+0000", 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6699999999999999e-07 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166268+0000", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.276e-06 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166293+0000", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.527e-05 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166311+0000", 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7439e-05 2024-09-11T05:51:08.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166434+0000", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012318299999999999 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166493+0000", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9083000000000003e-05 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.167936+0000", 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014426759999999999 2024-09-11T05:51:08.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.168038+0000", 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010211499999999999 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:7 2.2 e22/19 2:5cea7035:::rbd_info:head v 22'2, mlcod=0'0)", 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.693807926, 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019653880000000002, 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.870 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173227+0000", 2024-09-11T05:51:08.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6599999999999996e-07 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173241+0000", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4243999999999999e-05 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173267+0000", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6063000000000001e-05 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173297+0000", 2024-09-11T05:51:08.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0513000000000001e-05 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173470+0000", 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017282500000000001 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173553+0000", 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2637000000000002e-05 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.175108+0000", 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001555641 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.175191+0000", 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2799000000000005e-05 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:9 3.1a e26/23 3:5cea7035:::rbd_info:head v 26'2, mlcod=0'0)", 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:08.874 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.356550315, 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001489523, 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:08.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8899999999999999e-07 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510491+0000", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1400000000000002e-06 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510515+0000", 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4102999999999999e-05 2024-09-11T05:51:08.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510529+0000", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3208e-05 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510640+0000", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011099600000000001 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510668+0000", 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8379999999999999e-05 2024-09-11T05:51:08.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.511878+0000", 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001209711 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.511973+0000", 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5396000000000006e-05 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.878 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:08.879 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:08.879 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:08.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:08.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:08.967 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:08.966+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:08.996 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:08.996 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:08.996 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:08.996 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:09.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:09.051 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:09.050+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:09.054 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:09.055 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:09.074 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:09.074 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:09.091 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:09.091 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:09.108 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:09.108 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:09.128 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:09.129 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:09.144 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:09.146+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:09.149 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:09.149 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:09.161 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.161 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.169 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:09.169 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:09.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:09.191 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:09.191 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:09.251 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:09.250+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:09.299 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.299 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:09.359 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:09.358+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:09.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:09.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:09.401 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:51:09.401 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-11T05:51:09.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-09-11T05:51:09.447 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:09.446+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:09.476 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.476 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:09.504 INFO:teuthology.orchestra.run.smithi165.stdout:1 writesame 0xa45059d thru 0xa456d9b data_size 0x1(0x67ff bytes) 2024-09-11T05:51:09.505 INFO:teuthology.orchestra.run.smithi165.stdout:2 punch from 0x8a1cce0 to 0x8a203ec, (0x370c bytes) 2024-09-11T05:51:09.505 INFO:teuthology.orchestra.run.smithi165.stdout:3 writesame 0x97d7924 thru 0x97db73b data_size 0x4(0x3e18 bytes) 2024-09-11T05:51:09.510 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.511 INFO:teuthology.orchestra.run.smithi165.stdout:6 punch from 0x72920e0 to 0x7292509, (0x429 bytes) 2024-09-11T05:51:09.511 INFO:teuthology.orchestra.run.smithi165.stdout:9 writesame 0x4a7ad24 thru 0x4a88cf4 data_size 0x47(0xdfd1 bytes) 2024-09-11T05:51:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:10 punch from 0x6bdb97 to 0x6cd1fd, (0xf666 bytes) 2024-09-11T05:51:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:11 punch from 0x99e8aff to 0x99f75c2, (0xeac3 bytes) 2024-09-11T05:51:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0x4998e89 2024-09-11T05:51:09.525 INFO:teuthology.orchestra.run.smithi165.stdout:12 trunc from 0xa456d9c to 0x4998e89 2024-09-11T05:51:09.546 INFO:teuthology.orchestra.run.smithi165.stdout:14 punch from 0x3578780 to 0x357d962, (0x51e2 bytes) 2024-09-11T05:51:09.546 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0x71cdca9 2024-09-11T05:51:09.548 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:09.550+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:09.549 INFO:teuthology.orchestra.run.smithi165.stdout:15 trunc from 0x4998e89 to 0x71cdca9 2024-09-11T05:51:09.562 INFO:teuthology.orchestra.run.smithi165.stdout:16 punch from 0x2009b36 to 0x2018659, (0xeb23 bytes) 2024-09-11T05:51:09.575 INFO:teuthology.orchestra.run.smithi165.stdout:17 write 0x9718813 thru 0x971f7f3 (0x6fe1 bytes) 2024-09-11T05:51:09.580 INFO:teuthology.orchestra.run.smithi165.stdout:19 write 0x6966b66 thru 0x6968e04 (0x229f bytes) 2024-09-11T05:51:09.587 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0xc84ff7d 2024-09-11T05:51:09.590 INFO:teuthology.orchestra.run.smithi165.stdout:20 trunc from 0x971f7f4 to 0xc84ff7d 2024-09-11T05:51:09.615 INFO:teuthology.orchestra.run.smithi165.stdout:21 write 0xe172613 thru 0xe17334e (0xd3c bytes) 2024-09-11T05:51:09.623 INFO:teuthology.orchestra.run.smithi165.stdout:23 write 0x935000c thru 0x9353aea (0x3adf bytes) 2024-09-11T05:51:09.629 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.629 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.630 INFO:teuthology.orchestra.run.smithi165.stdout:28 punch from 0x1be2b2c to 0x1be8e2e, (0x6302 bytes) 2024-09-11T05:51:09.636 INFO:teuthology.orchestra.run.smithi165.stdout:30 trunc from 0xe17334f to 0xa3f4269 2024-09-11T05:51:09.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:09.649 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:09.646+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:09.656 INFO:teuthology.orchestra.run.smithi165.stdout:32 punch from 0xcea516 to 0xcf015c, (0x5c46 bytes) 2024-09-11T05:51:09.656 INFO:teuthology.orchestra.run.smithi165.stdout:34 write 0x75a89f8 thru 0x75afe93 (0x749c bytes) 2024-09-11T05:51:09.660 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.661 INFO:teuthology.orchestra.run.smithi165.stdout:37 write 0x7b67672 thru 0x7b6ca94 (0x5423 bytes) 2024-09-11T05:51:09.666 INFO:teuthology.orchestra.run.smithi165.stdout:38 writesame 0x865520e thru 0x86627e0 data_size 0x2b(0xd5d3 bytes) 2024-09-11T05:51:09.684 INFO:teuthology.orchestra.run.smithi165.stdout:39 writesame 0xcec0dbb thru 0xcecf311 data_size 0x1(0xe557 bytes) 2024-09-11T05:51:09.688 INFO:teuthology.orchestra.run.smithi165.stdout:40 trunc from 0xcecf312 to 0x9bc57e6 2024-09-11T05:51:09.698 INFO:teuthology.orchestra.run.smithi165.stdout:41 read 0x4f42492 thru 0x4f48ba8 (0x6717 bytes) 2024-09-11T05:51:09.698 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.698 INFO:teuthology.orchestra.run.smithi165.stdout:45 punch from 0x1912210 to 0x1921ff4, (0xfde4 bytes) 2024-09-11T05:51:09.705 INFO:teuthology.orchestra.run.smithi165.stdout:47 trunc from 0x9bc57e6 to 0x4835818 2024-09-11T05:51:09.728 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.728 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.729 INFO:teuthology.orchestra.run.smithi165.stdout:50 trunc from 0x4835818 to 0x56cc492 2024-09-11T05:51:09.741 INFO:teuthology.orchestra.run.smithi165.stdout:52 trunc from 0x56cc492 to 0x41bea2a 2024-09-11T05:51:09.749 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:09.746+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:09.761 INFO:teuthology.orchestra.run.smithi165.stdout:53 read 0x24cd879 thru 0x24d8d34 (0xb4bc bytes) 2024-09-11T05:51:09.761 INFO:teuthology.orchestra.run.smithi165.stdout:54 read 0xa04082 thru 0xa08932 (0x48b1 bytes) 2024-09-11T05:51:09.774 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.775 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.785 INFO:teuthology.orchestra.run.smithi165.stdout:55 writesame 0xeb52e74 thru 0xeb56ed6 data_size 0x1(0x4063 bytes) 2024-09-11T05:51:09.785 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0xcdf994c 2024-09-11T05:51:09.788 INFO:teuthology.orchestra.run.smithi165.stdout:56 trunc from 0xeb56ed7 to 0xcdf994c 2024-09-11T05:51:09.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:09.798 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.799 INFO:teuthology.orchestra.run.smithi165.stdout:58 punch from 0xc34a664 to 0xc34ae8d, (0x829 bytes) 2024-09-11T05:51:09.799 INFO:teuthology.orchestra.run.smithi165.stdout:59 write 0xb70c2de thru 0xb70f158 (0x2e7b bytes) 2024-09-11T05:51:09.803 INFO:teuthology.orchestra.run.smithi165.stdout:60 write 0x436010e thru 0x4366bc9 (0x6abc bytes) 2024-09-11T05:51:09.806 INFO:teuthology.orchestra.run.smithi165.stdout:61 write 0x2c834c6 thru 0x2c8743f (0x3f7a bytes) 2024-09-11T05:51:09.811 INFO:teuthology.orchestra.run.smithi165.stdout:62 trunc from 0xcdf994c to 0xbe933c7 2024-09-11T05:51:09.821 INFO:teuthology.orchestra.run.smithi165.stdout:63 write 0x5f204d6 thru 0x5f260d4 (0x5bff bytes) 2024-09-11T05:51:09.825 INFO:teuthology.orchestra.run.smithi165.stdout:65 read 0x6a0be58 thru 0x6a1bd94 (0xff3d bytes) 2024-09-11T05:51:09.825 INFO:teuthology.orchestra.run.smithi165.stdout:66 write 0x6342603 thru 0x634df4b (0xb949 bytes) 2024-09-11T05:51:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:67 punch from 0x38526e7 to 0x3854ff5, (0x290e bytes) 2024-09-11T05:51:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:70 read 0x19ce242 thru 0x19d332b (0x50ea bytes) 2024-09-11T05:51:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:72 write 0x6d4a765 thru 0x6d51df9 (0x7695 bytes) 2024-09-11T05:51:09.834 INFO:teuthology.orchestra.run.smithi165.stdout:73 punch from 0xba2e68d to 0xba3cd10, (0xe683 bytes) 2024-09-11T05:51:09.834 INFO:teuthology.orchestra.run.smithi165.stdout:75 punch from 0x139a7e7 to 0x139cd52, (0x256b bytes) 2024-09-11T05:51:09.841 INFO:teuthology.orchestra.run.smithi165.stdout:76 write 0xc295182 thru 0xc2a0274 (0xb0f3 bytes) 2024-09-11T05:51:09.846 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.846 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.846 INFO:teuthology.orchestra.run.smithi165.stdout:79 read 0xa22ca21 thru 0xa239e7c (0xd45c bytes) 2024-09-11T05:51:09.846 INFO:teuthology.orchestra.run.smithi165.stdout:80 punch from 0x5755654 to 0x5760269, (0xac15 bytes) 2024-09-11T05:51:09.846 INFO:teuthology.orchestra.run.smithi165.stdout:82 read 0x76c029 thru 0x76f482 (0x345a bytes) 2024-09-11T05:51:09.850 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:09.846+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:09.858 INFO:teuthology.orchestra.run.smithi165.stdout:83 writesame 0xdbbccca thru 0xdbbe100 data_size 0x2d(0x1437 bytes) 2024-09-11T05:51:09.858 INFO:teuthology.orchestra.run.smithi165.stdout:84 punch from 0xd34dd56 to 0xd35c190, (0xe43a bytes) 2024-09-11T05:51:09.858 INFO:teuthology.orchestra.run.smithi165.stdout:85 punch from 0x44760bc to 0x4485a2e, (0xf972 bytes) 2024-09-11T05:51:09.858 INFO:teuthology.orchestra.run.smithi165.stdout:87 read 0x8b984bf thru 0x8ba26e4 (0xa226 bytes) 2024-09-11T05:51:09.859 INFO:teuthology.orchestra.run.smithi165.stdout:88 read 0x906660d thru 0x906d705 (0x70f9 bytes) 2024-09-11T05:51:09.859 INFO:teuthology.orchestra.run.smithi165.stdout:89 writesame 0x2711aa6 thru 0x27145b7 data_size 0x25(0x2b12 bytes) 2024-09-11T05:51:09.864 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.864 INFO:teuthology.orchestra.run.smithi165.stdout:91 write 0xa4b2534 thru 0xa4bec00 (0xc6cd bytes) 2024-09-11T05:51:09.871 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.871 INFO:teuthology.orchestra.run.smithi165.stdout:95 write 0xaad616 thru 0xaba9c1 (0xd3ac bytes) 2024-09-11T05:51:09.877 INFO:teuthology.orchestra.run.smithi165.stdout:96 read 0x1d423f5 thru 0x1d494c4 (0x70d0 bytes) 2024-09-11T05:51:09.877 INFO:teuthology.orchestra.run.smithi165.stdout:97 write 0x3d8ca87 thru 0x3d93259 (0x67d3 bytes) 2024-09-11T05:51:09.883 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.883 INFO:teuthology.orchestra.run.smithi165.stdout:99 read 0xb49675 thru 0xb50cdb (0x7667 bytes) 2024-09-11T05:51:09.884 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:09.884 INFO:teuthology.orchestra.run.smithi165.stdout:101 write 0x9ccaa0b thru 0x9ccaa39 (0x2f bytes) 2024-09-11T05:51:09.889 INFO:teuthology.orchestra.run.smithi165.stdout:103 writesame 0xd9fd963 thru 0xda09bcb data_size 0x1(0xc269 bytes) 2024-09-11T05:51:09.927 INFO:teuthology.orchestra.run.smithi165.stdout:104 write 0x873028d thru 0x8732c9e (0x2a12 bytes) 2024-09-11T05:51:09.928 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:09.928 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:09.933 INFO:teuthology.orchestra.run.smithi165.stdout:105 write 0x592149a thru 0x592bb1b (0xa682 bytes) 2024-09-11T05:51:09.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:09.959 INFO:teuthology.orchestra.run.smithi165.stdout:107 trunc from 0xdbbe101 to 0x40ca1aa 2024-09-11T05:51:09.982 INFO:teuthology.orchestra.run.smithi165.stdout:108 read 0x38246c thru 0x383ba4 (0x1739 bytes) 2024-09-11T05:51:09.992 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:09.994+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:09.997 INFO:teuthology.orchestra.run.smithi165.stdout:109 writesame 0x63c0a0b thru 0x63c707d data_size 0x1(0x6673 bytes) 2024-09-11T05:51:09.997 INFO:teuthology.orchestra.run.smithi165.stdout:110 read 0xb4cd6d thru 0xb50a70 (0x3d04 bytes) 2024-09-11T05:51:10.025 INFO:teuthology.orchestra.run.smithi165.stdout:111 write 0xd8cdd7e thru 0xd8dbc76 (0xdef9 bytes) 2024-09-11T05:51:10.031 INFO:teuthology.orchestra.run.smithi165.stdout:112 read 0x51094b4 thru 0x5111ce0 (0x882d bytes) 2024-09-11T05:51:10.031 INFO:teuthology.orchestra.run.smithi165.stdout:114 read 0x4c2e83f thru 0x4c302ef (0x1ab1 bytes) 2024-09-11T05:51:10.032 INFO:teuthology.orchestra.run.smithi165.stdout:116 write 0xc435b48 thru 0xc4394ea (0x39a3 bytes) 2024-09-11T05:51:10.037 INFO:teuthology.orchestra.run.smithi165.stdout:117 write 0x8d88bd1 thru 0x8d94193 (0xb5c3 bytes) 2024-09-11T05:51:10.045 INFO:teuthology.orchestra.run.smithi165.stdout:119 writesame 0xd45b98e thru 0xd464dcb data_size 0x19(0x943e bytes) 2024-09-11T05:51:10.052 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:10.050+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:10.053 INFO:teuthology.orchestra.run.smithi165.stdout:120 writesame 0x5438a87 thru 0x543cc55 data_size 0x1(0x41cf bytes) 2024-09-11T05:51:10.075 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.075 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.079 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.079 INFO:teuthology.orchestra.run.smithi165.stdout:122 read 0xb74680 thru 0xb7b9b2 (0x7333 bytes) 2024-09-11T05:51:10.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:10.111 INFO:teuthology.orchestra.run.smithi165.stdout:125 trunc from 0xd8dbc77 to 0x1e31a66 2024-09-11T05:51:10.149 INFO:teuthology.orchestra.run.smithi165.stdout:126 writesame 0x3ab6ca9 thru 0x3abaf0a data_size 0x1d(0x4262 bytes) 2024-09-11T05:51:10.180 INFO:teuthology.orchestra.run.smithi165.stdout:127 write 0xa35d73f thru 0xa3611db (0x3a9d bytes) 2024-09-11T05:51:10.181 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:10.178+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:10.184 INFO:teuthology.orchestra.run.smithi165.stdout:128 read 0x90f6bab thru 0x9102f38 (0xc38e bytes) 2024-09-11T05:51:10.184 INFO:teuthology.orchestra.run.smithi165.stdout:130 write 0x731c107 thru 0x7324197 (0x8091 bytes) 2024-09-11T05:51:10.190 INFO:teuthology.orchestra.run.smithi165.stdout:131 writesame 0xf76584 thru 0xf7cd32 data_size 0x13(0x67af bytes) 2024-09-11T05:51:10.197 INFO:teuthology.orchestra.run.smithi165.stdout:134 read 0x30f11c1 thru 0x30f779c (0x65dc bytes) 2024-09-11T05:51:10.197 INFO:teuthology.orchestra.run.smithi165.stdout:135 writesame 0x575f787 thru 0x5769643 data_size 0x1(0x9ebd bytes) 2024-09-11T05:51:10.248 INFO:teuthology.orchestra.run.smithi165.stdout:137 read 0x9f9025f thru 0x9f9948c (0x922e bytes) 2024-09-11T05:51:10.257 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.258 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:10.273 INFO:teuthology.orchestra.run.smithi165.stdout:138 writesame 0xdb50db0 thru 0xdb58060 data_size 0x1(0x72b1 bytes) 2024-09-11T05:51:10.282 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:10.278+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:10.301 INFO:teuthology.orchestra.run.smithi165.stdout:139 trunc from 0xdb58061 to 0x23941a5 2024-09-11T05:51:10.322 INFO:teuthology.orchestra.run.smithi165.stdout:140 punch from 0x21560e6 to 0x215a217, (0x4131 bytes) 2024-09-11T05:51:10.323 INFO:teuthology.orchestra.run.smithi165.stdout:142 read 0x15920 thru 0x20e7b (0xb55c bytes) 2024-09-11T05:51:10.342 INFO:teuthology.orchestra.run.smithi165.stdout:144 write 0x5c60cc5 thru 0x5c612e0 (0x61c bytes) 2024-09-11T05:51:10.348 INFO:teuthology.orchestra.run.smithi165.stdout:147 read 0x540ce7a thru 0x540da3b (0xbc2 bytes) 2024-09-11T05:51:10.348 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.348 INFO:teuthology.orchestra.run.smithi165.stdout:149 read 0x420d21f thru 0x421763d (0xa41f bytes) 2024-09-11T05:51:10.349 INFO:teuthology.orchestra.run.smithi165.stdout:151 read 0x5752111 thru 0x575ec74 (0xcb64 bytes) 2024-09-11T05:51:10.349 INFO:teuthology.orchestra.run.smithi165.stdout:152 write 0x340dcba thru 0x341a344 (0xc68b bytes) 2024-09-11T05:51:10.356 INFO:teuthology.orchestra.run.smithi165.stdout:153 writesame 0x368e03f thru 0x3694b3f data_size 0x1(0x6b01 bytes) 2024-09-11T05:51:10.382 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:10.382+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:10.400 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.400 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.406 INFO:teuthology.orchestra.run.smithi165.stdout:155 write 0x7348e3f thru 0x734c551 (0x3713 bytes) 2024-09-11T05:51:10.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:10.423 INFO:teuthology.orchestra.run.smithi165.stdout:159 write 0x777eaba thru 0x778e9b0 (0xfef7 bytes) 2024-09-11T05:51:10.443 INFO:teuthology.orchestra.run.smithi165.stdout:160 write 0x895779a thru 0x896064b (0x8eb2 bytes) 2024-09-11T05:51:10.449 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0xcf32c88 2024-09-11T05:51:10.461 INFO:teuthology.orchestra.run.smithi165.stdout:161 trunc from 0x896064c to 0xcf32c88 2024-09-11T05:51:10.472 INFO:teuthology.orchestra.run.smithi165.stdout:162 read 0x9846e5 thru 0x9874e1 (0x2dfd bytes) 2024-09-11T05:51:10.473 INFO:teuthology.orchestra.run.smithi165.stdout:163 write 0x7b1e342 thru 0x7b2109f (0x2d5e bytes) 2024-09-11T05:51:10.479 INFO:teuthology.orchestra.run.smithi165.stdout:167 writesame 0xe8904e thru 0xe8d867 data_size 0xb(0x481a bytes) 2024-09-11T05:51:10.484 INFO:teuthology.orchestra.run.smithi165.stdout:168 punch from 0x34ee4e1 to 0x34fb9ee, (0xd50d bytes) 2024-09-11T05:51:10.484 INFO:teuthology.orchestra.run.smithi165.stdout:169 read 0x2d2fa02 thru 0x2d340d6 (0x46d5 bytes) 2024-09-11T05:51:10.499 INFO:teuthology.orchestra.run.smithi165.stdout:171 trunc from 0xcf32c88 to 0x77e542c 2024-09-11T05:51:10.504 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:10.506+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:10.523 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-11T05:51:10.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:51:10.531 INFO:teuthology.orchestra.run.smithi165.stdout:173 write 0x5040d83 thru 0x5040e31 (0xaf bytes) 2024-09-11T05:51:10.536 INFO:teuthology.orchestra.run.smithi165.stdout:174 read 0x4bff588 thru 0x4c0a257 (0xacd0 bytes) 2024-09-11T05:51:10.553 INFO:teuthology.orchestra.run.smithi165.stdout:176 writesame 0xc33b91e thru 0xc342c03 data_size 0x4d(0x72e6 bytes) 2024-09-11T05:51:10.553 INFO:teuthology.orchestra.run.smithi165.stdout:177 writesame 0x4870151 thru 0x4871eb3 data_size 0x1(0x1d63 bytes) 2024-09-11T05:51:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:179 read 0x3491b9b thru 0x349213d (0x5a3 bytes) 2024-09-11T05:51:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:180 read 0x2e09fef thru 0x2e13f41 (0x9f53 bytes) 2024-09-11T05:51:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:184 writesame 0xa74d2c2 thru 0xa754c0a data_size 0x1(0x7949 bytes) 2024-09-11T05:51:10.584 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:10.582+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:10.590 INFO:teuthology.orchestra.run.smithi165.stdout:185 write 0xa55721 thru 0xa65287 (0xfb67 bytes) 2024-09-11T05:51:10.593 INFO:teuthology.orchestra.run.smithi165.stdout:188 write 0x6a477e3 thru 0x6a4cd0f (0x552d bytes) 2024-09-11T05:51:10.596 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.596 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.597 INFO:teuthology.orchestra.run.smithi165.stdout:191 read 0x8ef1b43 thru 0x8eff3be (0xd87c bytes) 2024-09-11T05:51:10.609 INFO:teuthology.orchestra.run.smithi165.stdout:192 trunc from 0xc342c04 to 0x51876a6 2024-09-11T05:51:10.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:10.639 INFO:teuthology.orchestra.run.smithi165.stdout:193 write 0x8f61be6 thru 0x8f6e701 (0xcb1c bytes) 2024-09-11T05:51:10.643 INFO:teuthology.orchestra.run.smithi165.stdout:194 punch from 0x7a43447 to 0x7a48100, (0x4cb9 bytes) 2024-09-11T05:51:10.644 INFO:teuthology.orchestra.run.smithi165.stdout:195 read 0x2198113 thru 0x21a6811 (0xe6ff bytes) 2024-09-11T05:51:10.644 INFO:teuthology.orchestra.run.smithi165.stdout:196 punch from 0x84ed1c9 to 0x84ed2bd, (0xf4 bytes) 2024-09-11T05:51:10.655 INFO:teuthology.orchestra.run.smithi165.stdout:197 writesame 0xaafce3c thru 0xab06cb8 data_size 0x1(0x9e7d bytes) 2024-09-11T05:51:10.655 INFO:teuthology.orchestra.run.smithi165.stdout:198 writesame 0x2b17aac thru 0x2b2638e data_size 0x77(0xe8e3 bytes) 2024-09-11T05:51:10.659 INFO:teuthology.orchestra.run.smithi165.stdout:199 read 0x7d44e5b thru 0x7d52ea4 (0xe04a bytes) 2024-09-11T05:51:10.659 INFO:teuthology.orchestra.run.smithi165.stdout:201 punch from 0x5106ba0 to 0x510826f, (0x16cf bytes) 2024-09-11T05:51:10.659 INFO:teuthology.orchestra.run.smithi165.stdout:202 writesame 0x5d21f26 thru 0x5d28954 data_size 0x11(0x6a2f bytes) 2024-09-11T05:51:10.663 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.663 INFO:teuthology.orchestra.run.smithi165.stdout:205 read 0xa616ff9 thru 0xa623ecc (0xced4 bytes) 2024-09-11T05:51:10.664 INFO:teuthology.orchestra.run.smithi165.stdout:206 read 0x3c05b01 thru 0x3c0fe42 (0xa342 bytes) 2024-09-11T05:51:10.664 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.664 INFO:teuthology.orchestra.run.smithi165.stdout:209 writesame 0x8d628d3 thru 0x8d6a75c data_size 0x3(0x7e8a bytes) 2024-09-11T05:51:10.677 INFO:teuthology.orchestra.run.smithi165.stdout:210 write 0xbbabb2b thru 0xbbb4392 (0x8868 bytes) 2024-09-11T05:51:10.685 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:10.682+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:10.695 INFO:teuthology.orchestra.run.smithi165.stdout:211 trunc from 0xbbb4393 to 0x191e9bb 2024-09-11T05:51:10.707 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.707 INFO:teuthology.orchestra.run.smithi165.stdout:215 punch from 0xe7c978 to 0xe83696, (0x6d1e bytes) 2024-09-11T05:51:10.707 INFO:teuthology.orchestra.run.smithi165.stdout:216 read 0x552b64 thru 0x55d4ec (0xa989 bytes) 2024-09-11T05:51:10.711 INFO:teuthology.orchestra.run.smithi165.stdout:217 trunc from 0x191e9bb to 0x3ed9117 2024-09-11T05:51:10.716 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.734 INFO:teuthology.orchestra.run.smithi165.stdout:220 write 0xbf9cc97 thru 0xbfa8a49 (0xbdb3 bytes) 2024-09-11T05:51:10.737 INFO:teuthology.orchestra.run.smithi165.stdout:223 writesame 0x41f5863 thru 0x4200ebd data_size 0x1b(0xb65b bytes) 2024-09-11T05:51:10.742 INFO:teuthology.orchestra.run.smithi165.stdout:225 writesame 0x33acef0 thru 0x33b5f02 data_size 0xb(0x9013 bytes) 2024-09-11T05:51:10.747 INFO:teuthology.orchestra.run.smithi165.stdout:226 punch from 0x96169e5 to 0x9620f83, (0xa59e bytes) 2024-09-11T05:51:10.747 INFO:teuthology.orchestra.run.smithi165.stdout:227 read 0x6d4ae59 thru 0x6d4edda (0x3f82 bytes) 2024-09-11T05:51:10.747 INFO:teuthology.orchestra.run.smithi165.stdout:229 read 0xa6a4d9e thru 0xa6a9240 (0x44a3 bytes) 2024-09-11T05:51:10.747 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.747 INFO:teuthology.orchestra.run.smithi165.stdout:232 writesame 0x68e77e1 thru 0x68e8f8d data_size 0x13(0x17ad bytes) 2024-09-11T05:51:10.750 INFO:teuthology.orchestra.run.smithi165.stdout:235 read 0x1816a7c thru 0x1825893 (0xee18 bytes) 2024-09-11T05:51:10.750 INFO:teuthology.orchestra.run.smithi165.stdout:236 read 0x6f570d9 thru 0x6f61fab (0xaed3 bytes) 2024-09-11T05:51:10.750 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.751 INFO:teuthology.orchestra.run.smithi165.stdout:241 write 0x82b3c7d thru 0x82b580d (0x1b91 bytes) 2024-09-11T05:51:10.754 INFO:teuthology.orchestra.run.smithi165.stdout:242 write 0x52e314f thru 0x52e5964 (0x2816 bytes) 2024-09-11T05:51:10.758 INFO:teuthology.orchestra.run.smithi165.stdout:244 write 0x9dc3fcf thru 0x9dd232f (0xe361 bytes) 2024-09-11T05:51:10.763 INFO:teuthology.orchestra.run.smithi165.stdout:246 write 0x883a988 thru 0x8847432 (0xcaab bytes) 2024-09-11T05:51:10.768 INFO:teuthology.orchestra.run.smithi165.stdout:247 writesame 0x7fb1e7d thru 0x7fbfd09 data_size 0x7(0xde8d bytes) 2024-09-11T05:51:10.777 INFO:teuthology.orchestra.run.smithi165.stdout:248 write 0x1ea1fab thru 0x1eae9d8 (0xca2e bytes) 2024-09-11T05:51:10.782 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.782 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.783 INFO:teuthology.orchestra.run.smithi165.stdout:249 read 0x99a5534 thru 0x99af079 (0x9b46 bytes) 2024-09-11T05:51:10.785 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:10.786+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:10.794 INFO:teuthology.orchestra.run.smithi165.stdout:250 write 0xdab515e thru 0xdabc699 (0x753c bytes) 2024-09-11T05:51:10.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:10.800 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:10.801 INFO:teuthology.orchestra.run.smithi165.stdout:253 read 0x2563bbe thru 0x256e30b (0xa74e bytes) 2024-09-11T05:51:10.815 INFO:teuthology.orchestra.run.smithi165.stdout:254 trunc from 0xdabc69a to 0x42a06f7 2024-09-11T05:51:10.842 INFO:teuthology.orchestra.run.smithi165.stdout:255 trunc from 0x42a06f7 to 0x146076 2024-09-11T05:51:10.861 INFO:teuthology.orchestra.run.smithi165.stdout:256 read 0xfc38a thru 0x10821c (0xbe93 bytes) 2024-09-11T05:51:10.861 INFO:teuthology.orchestra.run.smithi165.stdout:257 punch from 0xac467 to 0xaf226, (0x2dbf bytes) 2024-09-11T05:51:10.888 INFO:teuthology.orchestra.run.smithi165.stdout:262 write 0xd6253cb thru 0xd6266a3 (0x12d9 bytes) 2024-09-11T05:51:10.889 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:10.890+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:10.893 INFO:teuthology.orchestra.run.smithi165.stdout:264 read 0x9cd62e6 thru 0x9ce31e6 (0xcf01 bytes) 2024-09-11T05:51:10.902 INFO:teuthology.orchestra.run.smithi165.stdout:265 trunc from 0xd6266a4 to 0x6c1ad35 2024-09-11T05:51:10.919 INFO:teuthology.orchestra.run.smithi165.stdout:266 write 0x1c7ef84 thru 0x1c83d2c (0x4da9 bytes) 2024-09-11T05:51:10.924 INFO:teuthology.orchestra.run.smithi165.stdout:267 read 0x1d4fed thru 0x1da54b (0x555f bytes) 2024-09-11T05:51:10.924 INFO:teuthology.orchestra.run.smithi165.stdout:269 punch from 0x362c81e to 0x3633af1, (0x72d3 bytes) 2024-09-11T05:51:10.924 INFO:teuthology.orchestra.run.smithi165.stdout:270 writesame 0x1963ae1 thru 0x196421f data_size 0x7(0x73f bytes) 2024-09-11T05:51:10.935 INFO:teuthology.orchestra.run.smithi165.stdout:274 trunc from 0x6c1ad35 to 0x9c3b3d8 2024-09-11T05:51:10.945 INFO:teuthology.orchestra.run.smithi165.stdout:275 read 0x5ea02d8 thru 0x5ea64b1 (0x61da bytes) 2024-09-11T05:51:10.970 INFO:teuthology.orchestra.run.smithi165.stdout:276 writesame 0xe9c3268 thru 0xe9cf5f5 data_size 0x1(0xc38e bytes) 2024-09-11T05:51:10.970 INFO:teuthology.orchestra.run.smithi165.stdout:277 punch from 0xb41a725 to 0xb41ff4e, (0x5829 bytes) 2024-09-11T05:51:10.970 INFO:teuthology.orchestra.run.smithi165.stdout:278 writesame 0x5ec44d9 thru 0x5ec7279 data_size 0x1(0x2da1 bytes) 2024-09-11T05:51:10.972 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:10.972 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:10.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:10.987 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:10.986+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:10.997 INFO:teuthology.orchestra.run.smithi165.stdout:280 writesame 0xebbeb8f thru 0xebcc452 data_size 0x2(0xd8c4 bytes) 2024-09-11T05:51:11.006 INFO:teuthology.orchestra.run.smithi165.stdout:281 trunc from 0xebcc453 to 0xad54d41 2024-09-11T05:51:11.031 INFO:teuthology.orchestra.run.smithi165.stdout:283 trunc from 0xad54d41 to 0x487cb50 2024-09-11T05:51:11.078 INFO:teuthology.orchestra.run.smithi165.stdout:285 write 0xbe86090 thru 0xbe8d71e (0x768f bytes) 2024-09-11T05:51:11.085 INFO:teuthology.orchestra.run.smithi165.stdout:286 punch from 0x3107427 to 0x310a016, (0x2bef bytes) 2024-09-11T05:51:11.085 INFO:teuthology.orchestra.run.smithi165.stdout:288 write 0x997c678 thru 0x99879d6 (0xb35f bytes) 2024-09-11T05:51:11.087 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:11.086+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:11.093 INFO:teuthology.orchestra.run.smithi165.stdout:291 writesame 0xf982ff thru 0xf9fd28 data_size 0x13(0x7a2a bytes) 2024-09-11T05:51:11.101 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.101 INFO:teuthology.orchestra.run.smithi165.stdout:293 write 0x23da859 thru 0x23e70ca (0xc872 bytes) 2024-09-11T05:51:11.107 INFO:teuthology.orchestra.run.smithi165.stdout:294 write 0x96f5496 thru 0x97002ba (0xae25 bytes) 2024-09-11T05:51:11.114 INFO:teuthology.orchestra.run.smithi165.stdout:295 read 0x56ce08a thru 0x56ce5f9 (0x570 bytes) 2024-09-11T05:51:11.115 INFO:teuthology.orchestra.run.smithi165.stdout:296 read 0x88a7a8 thru 0x898d5f (0xe5b8 bytes) 2024-09-11T05:51:11.115 INFO:teuthology.orchestra.run.smithi165.stdout:297 punch from 0x37c87ab to 0x37d59cb, (0xd220 bytes) 2024-09-11T05:51:11.115 INFO:teuthology.orchestra.run.smithi165.stdout:298 write 0x98bc025 thru 0x98c85a0 (0xc57c bytes) 2024-09-11T05:51:11.118 INFO:teuthology.orchestra.run.smithi165.stdout:300 writesame 0x6a05d88 thru 0x6a0c6e3 data_size 0x16(0x695c bytes) 2024-09-11T05:51:11.126 INFO:teuthology.orchestra.run.smithi165.stdout:301 read 0xb496de2 thru 0xb49ca9b (0x5cba bytes) 2024-09-11T05:51:11.126 INFO:teuthology.orchestra.run.smithi165.stdout:302 read 0x57f561d thru 0x57fa691 (0x5075 bytes) 2024-09-11T05:51:11.126 INFO:teuthology.orchestra.run.smithi165.stdout:304 read 0x5a1b3a2 thru 0x5a2309e (0x7cfd bytes) 2024-09-11T05:51:11.126 INFO:teuthology.orchestra.run.smithi165.stdout:305 write 0x4a9c923 thru 0x4aa9512 (0xcbf0 bytes) 2024-09-11T05:51:11.133 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.133 INFO:teuthology.orchestra.run.smithi165.stdout:308 read 0x30f3896 thru 0x3101af4 (0xe25f bytes) 2024-09-11T05:51:11.133 INFO:teuthology.orchestra.run.smithi165.stdout:309 punch from 0xa62ab6a to 0xa639abd, (0xef53 bytes) 2024-09-11T05:51:11.136 INFO:teuthology.orchestra.run.smithi165.stdout:310 trunc from 0xbe8d71f to 0xcca6f65 2024-09-11T05:51:11.148 INFO:teuthology.orchestra.run.smithi165.stdout:311 read 0x6d14ad9 thru 0x6d19989 (0x4eb1 bytes) 2024-09-11T05:51:11.148 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.149 INFO:teuthology.orchestra.run.smithi165.stdout:313 write 0x2345c24 thru 0x2354dfc (0xf1d9 bytes) 2024-09-11T05:51:11.152 INFO:teuthology.orchestra.run.smithi165.stdout:314 read 0x1ac8afb thru 0x1ad5bca (0xd0d0 bytes) 2024-09-11T05:51:11.153 INFO:teuthology.orchestra.run.smithi165.stdout:315 writesame 0x8612942 thru 0x8614a88 data_size 0x1(0x2147 bytes) 2024-09-11T05:51:11.159 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:11.159 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:11.168 INFO:teuthology.orchestra.run.smithi165.stdout:316 punch from 0x2a29729 to 0x2a31ff2, (0x88c9 bytes) 2024-09-11T05:51:11.168 INFO:teuthology.orchestra.run.smithi165.stdout:317 write 0x89e48c2 thru 0x89eba8c (0x71cb bytes) 2024-09-11T05:51:11.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:11.175 INFO:teuthology.orchestra.run.smithi165.stdout:318 punch from 0x9110a5a to 0x9119858, (0x8dfe bytes) 2024-09-11T05:51:11.175 INFO:teuthology.orchestra.run.smithi165.stdout:319 read 0x771929a thru 0x7719f6f (0xcd6 bytes) 2024-09-11T05:51:11.175 INFO:teuthology.orchestra.run.smithi165.stdout:320 write 0x93e6790 thru 0x93edd9d (0x760e bytes) 2024-09-11T05:51:11.182 INFO:teuthology.orchestra.run.smithi165.stdout:321 punch from 0x7c7c0d6 to 0x7c7f924, (0x384e bytes) 2024-09-11T05:51:11.201 INFO:teuthology.orchestra.run.smithi165.stdout:323 trunc from 0xcca6f65 to 0x5f9a5bf 2024-09-11T05:51:11.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:11.222 INFO:teuthology.orchestra.run.smithi165.stdout:324 read 0x4a02dc thru 0x4a1317 (0x103c bytes) 2024-09-11T05:51:11.224 INFO:teuthology.orchestra.run.smithi165.stdout:326 trunc from 0x5f9a5bf to 0x6831dd5 2024-09-11T05:51:11.224 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:11.226+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:11.231 INFO:teuthology.orchestra.run.smithi165.stdout:327 read 0x5dc903e thru 0x5dcccec (0x3caf bytes) 2024-09-11T05:51:11.263 INFO:teuthology.orchestra.run.smithi165.stdout:328 writesame 0xeb5b8cd thru 0xeb64ff9 data_size 0xd(0x972d bytes) 2024-09-11T05:51:11.263 INFO:teuthology.orchestra.run.smithi165.stdout:330 read 0xdf69817 thru 0xdf77829 (0xe013 bytes) 2024-09-11T05:51:11.289 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:11.290+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:11.295 INFO:teuthology.orchestra.run.smithi165.stdout:335 trunc from 0xeb64ffa to 0xc6322 2024-09-11T05:51:11.303 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:11.303 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:11.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:11.319 INFO:teuthology.orchestra.run.smithi165.stdout:336 punch from 0x424b9 to 0x492ad, (0x6df4 bytes) 2024-09-11T05:51:11.339 INFO:teuthology.orchestra.run.smithi165.stdout:338 write 0x46e2443 thru 0x46e76f4 (0x52b2 bytes) 2024-09-11T05:51:11.361 INFO:teuthology.orchestra.run.smithi165.stdout:339 write 0xac31175 thru 0xac3c7a4 (0xb630 bytes) 2024-09-11T05:51:11.366 INFO:teuthology.orchestra.run.smithi165.stdout:341 read 0x230e050 thru 0x2312661 (0x4612 bytes) 2024-09-11T05:51:11.366 INFO:teuthology.orchestra.run.smithi165.stdout:342 read 0x8864e44 thru 0x886f115 (0xa2d2 bytes) 2024-09-11T05:51:11.366 INFO:teuthology.orchestra.run.smithi165.stdout:343 write 0x629243a thru 0x629b5d0 (0x9197 bytes) 2024-09-11T05:51:11.370 INFO:teuthology.orchestra.run.smithi165.stdout:344 punch from 0x988de85 to 0x9894d7d, (0x6ef8 bytes) 2024-09-11T05:51:11.370 INFO:teuthology.orchestra.run.smithi165.stdout:345 read 0x5e98ae thru 0x5eb018 (0x176b bytes) 2024-09-11T05:51:11.370 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.372 INFO:teuthology.orchestra.run.smithi165.stdout:348 trunc from 0xac3c7a5 to 0x9969613 2024-09-11T05:51:11.376 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:11.376 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:11.383 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0xe9a75f0 2024-09-11T05:51:11.391 INFO:teuthology.orchestra.run.smithi165.stdout:350 trunc from 0x9969613 to 0xe9a75f0 2024-09-11T05:51:11.394 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:11.394 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:11.401 INFO:teuthology.orchestra.run.smithi165.stdout:truncating to largest ever: 0xee60f43 2024-09-11T05:51:11.401 INFO:teuthology.orchestra.run.smithi165.stdout:352 trunc from 0xe9a75f0 to 0xee60f43 2024-09-11T05:51:11.404 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:11.406+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:11.405 INFO:teuthology.orchestra.run.smithi165.stdout:354 punch from 0x42ccc3c to 0x42dc87f, (0xfc43 bytes) 2024-09-11T05:51:11.406 INFO:teuthology.orchestra.run.smithi165.stdout:355 punch from 0xebd8554 to 0xebe2b7e, (0xa62a bytes) 2024-09-11T05:51:11.406 INFO:teuthology.orchestra.run.smithi165.stdout:356 write 0xeb5efdc thru 0xeb6150e (0x2533 bytes) 2024-09-11T05:51:11.409 INFO:teuthology.orchestra.run.smithi165.stdout:357 read 0x2fdd5a5 thru 0x2fe32d7 (0x5d33 bytes) 2024-09-11T05:51:11.410 INFO:teuthology.orchestra.run.smithi165.stdout:358 write 0xc2cf329 thru 0xc2d1cd7 (0x29af bytes) 2024-09-11T05:51:11.411 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:11.411 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:11.413 INFO:teuthology.orchestra.run.smithi165.stdout:359 writesame 0x9b07334 thru 0x9b16e72 data_size 0x1(0xfb3f bytes) 2024-09-11T05:51:11.437 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:11.437 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:11.447 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.447 INFO:teuthology.orchestra.run.smithi165.stdout:362 read 0x63049ee thru 0x630f39a (0xa9ad bytes) 2024-09-11T05:51:11.448 INFO:teuthology.orchestra.run.smithi165.stdout:364 read 0x65cb55e thru 0x65cf812 (0x42b5 bytes) 2024-09-11T05:51:11.448 INFO:teuthology.orchestra.run.smithi165.stdout:365 write 0x62d5a2c thru 0x62decb1 (0x9286 bytes) 2024-09-11T05:51:11.451 INFO:teuthology.orchestra.run.smithi165.stdout:366 read 0xe722d42 thru 0xe72dd9a (0xb059 bytes) 2024-09-11T05:51:11.452 INFO:teuthology.orchestra.run.smithi165.stdout:367 read 0x87e63d0 thru 0x87eef65 (0x8b96 bytes) 2024-09-11T05:51:11.452 INFO:teuthology.orchestra.run.smithi165.stdout:370 read 0x9f94d5b thru 0x9f9a51e (0x57c4 bytes) 2024-09-11T05:51:11.452 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:11.458 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:11.458 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:11.469 INFO:teuthology.orchestra.run.smithi165.stdout:375 trunc from 0xee60f43 to 0x240a071 2024-09-11T05:51:11.481 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:11.481 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:11.490 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:11.486+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:11.491 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:11.491 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:11.491 INFO:teuthology.orchestra.run.smithi165.stdout:376 read 0x35741 thru 0x42e6c (0xd72c bytes) 2024-09-11T05:51:11.491 INFO:teuthology.orchestra.run.smithi165.stdout:377 punch from 0x9dfab6 to 0x9e13ba, (0x1904 bytes) 2024-09-11T05:51:11.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:11.508 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:11.508 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:11.515 INFO:teuthology.orchestra.run.smithi165.stdout:379 write 0xc8d48d3 thru 0xc8e2416 (0xdb44 bytes) 2024-09-11T05:51:11.520 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:51:11.526 INFO:teuthology.orchestra.run.smithi165.stdout:380 read 0x17a645b thru 0x17a7dd1 (0x1977 bytes) 2024-09-11T05:51:11.526 INFO:teuthology.orchestra.run.smithi165.stdout:381 clone 1 order 20 su 32768 sc 9 2024-09-11T05:51:11.530 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:11.531 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:11.592 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:11.594+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:11.666 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:11.667 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:11.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:11.692 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:11.690+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:11.793 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:11.794+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.667685683999999, 2024-09-11T05:51:11.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011567851000000001, 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179401+0000", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.573e-06 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179413+0000", 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2670000000000001e-05 2024-09-11T05:51:11.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179469+0000", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5903000000000003e-05 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179485+0000", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.186952+0000", 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074664290000000001 2024-09-11T05:51:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187123+0000", 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017143900000000001 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187336+0000", 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000212692 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190287+0000", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029509169999999999 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190763+0000", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000475799 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190894+0000", 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013149599999999999 2024-09-11T05:51:11.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190967+0000", 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2682999999999995e-05 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.667570022, 2024-09-11T05:51:11.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011710134000000001, 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179516+0000", 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179529+0000", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2204e-05 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179555+0000", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6602999999999999e-05 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179576+0000", 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0636999999999999e-05 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187937+0000", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083613920000000005 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188065+0000", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127532 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188273+0000", 2024-09-11T05:51:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020859200000000001 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190983+0000", 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027099189999999999 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191042+0000", 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9008999999999997e-05 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191159+0000", 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116556 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191225+0000", 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6060000000000001e-05 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.571872823, 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011718055999999999, 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 87 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275213+0000", 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7300000000000005e-07 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275308+0000", 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5353000000000007e-05 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275320+0000", 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1746000000000001e-05 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275328+0000", 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2770000000000005e-06 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.282992+0000", 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007663998 2024-09-11T05:51:11.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283142+0000", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015011899999999999 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283361+0000", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000218488 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286081+0000", 2024-09-11T05:51:11.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027206890000000001 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286782+0000", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070088400000000003 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286863+0000", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1225999999999998e-05 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286930+0000", 2024-09-11T05:51:11.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6502999999999997e-05 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.571737748, 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013655390999999999, 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 88 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:11.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275348+0000", 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275354+0000", 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.947e-06 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275367+0000", 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952999999999999e-05 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275375+0000", 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2230000000000007e-06 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284632+0000", 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092572620000000005 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284856+0000", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022421799999999999 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.285198+0000", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034152800000000001 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.287353+0000", 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021548800000000001 2024-09-11T05:51:11.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288869+0000", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001515728 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288944+0000", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5339999999999994e-05 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.289002+0000", 2024-09-11T05:51:11.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8471999999999999e-05 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410486468999999, 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013083005999999999, 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436600+0000", 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.553e-06 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436737+0000", 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436759+0000", 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1946e-05 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436775+0000", 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5767e-05 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444423+0000", 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007648874 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444555+0000", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013180900000000001 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444751+0000", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019567200000000001 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.446939+0000", 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021876830000000002 2024-09-11T05:51:11.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449422+0000", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002483791 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449562+0000", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013971900000000001 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449681+0000", 2024-09-11T05:51:11.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119126 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410279031000002, 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014821173, 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.880 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:11.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436807+0000", 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.637e-06 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436819+0000", 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1663e-05 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436866+0000", 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6879999999999998e-05 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436885+0000", 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9483e-05 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447203+0000", 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010317244 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447333+0000", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013070200000000001 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447532+0000", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019884199999999999 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451135+0000", 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036030820000000001 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451358+0000", 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022260800000000001 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451488+0000", 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130273 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451627+0000", 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013875899999999999 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410153155, 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017261187000000001, 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436933+0000", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436945+0000", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1939999999999999e-05 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436966+0000", 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0979999999999999e-05 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436981+0000", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5353e-05 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448139+0000", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011157373999999999 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448357+0000", 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021852500000000001 2024-09-11T05:51:11.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448717+0000", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451314+0000", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025974359999999998 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453934+0000", 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026193599999999998 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454076+0000", 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014191599999999999 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454193+0000", 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117199 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410075128999999, 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017801221999999998, 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:11.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437011+0000", 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3999999999999999e-06 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437151+0000", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140359 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437177+0000", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437204+0000", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7645999999999999e-05 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449807+0000", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012602197000000001 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449944+0000", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013775500000000001 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.450149+0000", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020427499999999999 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454334+0000", 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041850480000000002 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454533+0000", 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000198968 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454697+0000", 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164446 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454811+0000", 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113775 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.409848930999999, 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017714963, 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437237+0000", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5570000000000001e-06 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437249+0000", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1800000000000001e-05 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437278+0000", 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8866000000000001e-05 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437295+0000", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7447000000000001e-05 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451717+0000", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014421237999999999 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451859+0000", 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014248900000000001 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452057+0000", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019812200000000001 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454491+0000", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024333940000000002 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454830+0000", 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033955500000000001 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454900+0000", 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0049000000000006e-05 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454951+0000", 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0445999999999998e-05 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.902 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.409763325, 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019404264000000001, 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437323+0000", 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437333+0000", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0499999999999999e-05 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437356+0000", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2293000000000002e-05 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437369+0000", 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.381e-05 2024-09-11T05:51:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452677+0000", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015307215000000001 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452924+0000", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024779199999999999 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453257+0000", 2024-09-11T05:51:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000332944 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456421+0000", 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003164072 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456460+0000", 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8869999999999999e-05 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456617+0000", 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156305 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456726+0000", 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108993 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.409644343, 2024-09-11T05:51:11.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021955514999999998, 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437441+0000", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8999999999999995e-07 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437529+0000", 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8226999999999994e-05 2024-09-11T05:51:11.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437556+0000", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6806000000000001e-05 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437570+0000", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4003e-05 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455042+0000", 2024-09-11T05:51:11.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017471738000000001 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455198+0000", 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455397+0000", 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019827199999999999 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459080+0000", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036837860000000001 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459121+0000", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0589000000000001e-05 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459260+0000", 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013900199999999999 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459396+0000", 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135933 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.409498603999999, 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022184125999999998, 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437587+0000", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3699999999999999e-07 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437593+0000", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3829999999999997e-06 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437606+0000", 2024-09-11T05:51:11.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2649999999999999e-05 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437613+0000", 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2570000000000001e-06 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456824+0000", 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019210188999999999 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456953+0000", 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012939200000000001 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457153+0000", 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000199891 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459412+0000", 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022590000000000002 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459524+0000", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111716 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459657+0000", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013367199999999999 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459770+0000", 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113039 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.409459504000001, 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022423287, 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.920 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437626+0000", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2399999999999997e-07 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437632+0000", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0599999999999996e-06 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437642+0000", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5699999999999999e-06 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437650+0000", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4689999999999997e-06 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457786+0000", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020135574999999999 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458013+0000", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022734200000000001 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458415+0000", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040240300000000002 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459798+0000", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013821759999999999 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459914+0000", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116773 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459992+0000", 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8059000000000006e-05 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.460048+0000", 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6035999999999997e-05 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.931625767, 2024-09-11T05:51:11.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013428412000000001, 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915460+0000", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4759999999999999e-06 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915673+0000", 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021259499999999999 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915717+0000", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4206999999999999e-05 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915735+0000", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8073000000000001e-05 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.921996+0000", 2024-09-11T05:51:11.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062606429999999998 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922091+0000", 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4765000000000005e-05 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922203+0000", 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112596 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926133+0000", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003930022 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928670+0000", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025363069999999998 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928761+0000", 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1853000000000003e-05 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928887+0000", 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012587500000000001 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.931314499999999, 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013236257, 2024-09-11T05:51:11.931 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915772+0000", 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6559999999999999e-06 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915784+0000", 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2034e-05 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915809+0000", 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4839999999999999e-05 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915829+0000", 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9809999999999998e-05 2024-09-11T05:51:11.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923686+0000", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078569520000000004 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923818+0000", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132182 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924042+0000", 2024-09-11T05:51:11.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022383799999999999 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926144+0000", 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002102348 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928947+0000", 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028030070000000002 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928981+0000", 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4014999999999999e-05 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929006+0000", 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5575000000000002e-05 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.931228320999999, 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013436350999999999, 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915858+0000", 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5e-06 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915869+0000", 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1039999999999999e-05 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915890+0000", 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1056999999999998e-05 2024-09-11T05:51:11.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915907+0000", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6898999999999999e-05 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924934+0000", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090266600000000006 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925029+0000", 2024-09-11T05:51:11.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5313e-05 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925151+0000", 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122416 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926154+0000", 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010029559999999999 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929245+0000", 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003090939 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929272+0000", 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6843999999999999e-05 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929293+0000", 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0727e-05 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.931106868000001, 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016090376999999999, 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.944 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915979+0000", 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.623e-06 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916191+0000", 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000211385 2024-09-11T05:51:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916218+0000", 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7503000000000001e-05 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916242+0000", 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3657000000000001e-05 2024-09-11T05:51:11.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926301+0000", 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010058958999999999 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926430+0000", 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129472 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926644+0000", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000213332 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929041+0000", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023971750000000001 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932021+0000", 2024-09-11T05:51:11.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029796509999999998 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932048+0000", 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240999999999999e-05 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932068+0000", 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0378999999999999e-05 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.930797081000001, 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016382369000000001, 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916289+0000", 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.747e-06 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916301+0000", 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.201e-05 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916325+0000", 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3323e-05 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916364+0000", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9069999999999997e-05 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.927959+0000", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011595032 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928047+0000", 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7719999999999994e-05 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928196+0000", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149276 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929059+0000", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086328399999999997 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932625+0000", 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035663819999999999 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932651+0000", 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5732e-05 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932670+0000", 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8793000000000001e-05 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.196897761999999, 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011210540999999999, 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 158 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650188+0000", 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4300000000000001e-06 2024-09-11T05:51:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650427+0000", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023869200000000001 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650453+0000", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6026000000000001e-05 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650475+0000", 2024-09-11T05:51:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.145e-05 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657215+0000", 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067406150000000001 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657330+0000", 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114809 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657573+0000", 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243241 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660296+0000", 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027227359999999999 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661256+0000", 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00095981899999999999 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661329+0000", 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2819999999999997e-05 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661397+0000", 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8903000000000001e-05 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.196577761, 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013818587, 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 159 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650509+0000", 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.773e-06 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650520+0000", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1202999999999999e-05 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650544+0000", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4023999999999999e-05 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650576+0000", 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1922999999999999e-05 2024-09-11T05:51:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659257+0000", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086811560000000006 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659395+0000", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013841299999999999 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659593+0000", 2024-09-11T05:51:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197748 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660722+0000", 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011288680000000001 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664169+0000", 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034466929999999998 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:11.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664260+0000", 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0946000000000006e-05 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664326+0000", 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5840000000000007e-05 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:11.968 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:11.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:11.970 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:11.970+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:12.071 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:12.070+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:12.102 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:12.102 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:12.102 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:12.103 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:12.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:12.207 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:12.206+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:12.272 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:12.270+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:12.284 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:12.284 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:12.284 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:12.284 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:12.284 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:12.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:12.373 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:12.374+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:12.473 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:35 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'21, mlcod=17'16)", 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.286503473, 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003261576, 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185180+0000", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185181+0000", 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5200000000000001e-06 2024-09-11T05:51:12.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185282+0000", 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101363 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185303+0000", 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0193000000000001e-05 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185314+0000", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0923e-05 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185425+0000", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111203 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.185454+0000", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8955999999999999e-05 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.188334+0000", 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028801719999999999 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.188441+0000", 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107246 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:54 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'32, mlcod=17'29)", 2024-09-11T05:51:12.478 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.250413244000001, 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002724232, 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:12.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221270+0000", 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221272+0000", 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221289+0000", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7410000000000001e-05 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221325+0000", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5583999999999997e-05 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221344+0000", 2024-09-11T05:51:12.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9599e-05 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221449+0000", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104393 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.221494+0000", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5636000000000001e-05 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.223907+0000", 2024-09-11T05:51:12.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024125209999999999 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.223994+0000", 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7072999999999998e-05 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:83 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'53, mlcod=17'50)", 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:12.483 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.195771556, 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028129919999999998, 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275912+0000", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275913+0000", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.147e-06 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275922+0000", 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.713e-06 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275940+0000", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7989999999999999e-05 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.275951+0000", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.276061+0000", 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109722 2024-09-11T05:51:12.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.276137+0000", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5887e-05 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.278635+0000", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024986259999999999 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.278724+0000", 2024-09-11T05:51:12.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9086999999999996e-05 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'54)", 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.188890976, 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027993319999999999, 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.488 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282792+0000", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282793+0000", 2024-09-11T05:51:12.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5300000000000002e-07 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282854+0000", 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1416999999999998e-05 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282875+0000", 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0387e-05 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282885+0000", 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0373e-05 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.282988+0000", 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102539 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.283041+0000", 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2958999999999999e-05 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.285504+0000", 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024628279999999998 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.285591+0000", 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7875999999999997e-05 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.492 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:129 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'87, mlcod=17'85)", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.558281216000001, 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028747239999999999, 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913402+0000", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913404+0000", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1859999999999999e-06 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913421+0000", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6917000000000001e-05 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913459+0000", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.773e-05 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913478+0000", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9276e-05 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913625+0000", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014670899999999999 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.913681+0000", 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5983000000000003e-05 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916208+0000", 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025273230000000002 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916277+0000", 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.86e-05 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:131 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'89, mlcod=17'85)", 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.556692922, 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028937379999999999, 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914990+0000", 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.914992+0000", 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3799999999999999e-06 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915148+0000", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156859 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915172+0000", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.404e-05 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915185+0000", 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2086000000000001e-05 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915261+0000", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6342999999999997e-05 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.915290+0000", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9390000000000002e-05 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917784+0000", 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024937869999999999 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917884+0000", 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9852999999999994e-05 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:132 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'90, mlcod=17'85)", 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.554927962000001, 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053327920000000003, 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.501 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916755+0000", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916757+0000", 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4100000000000001e-06 2024-09-11T05:51:12.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916767+0000", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0496000000000001e-05 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916790+0000", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.315e-05 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916803+0000", 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2860000000000001e-05 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.916934+0000", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013144900000000001 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.917013+0000", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8076000000000003e-05 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922020+0000", 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050075650000000003 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922088+0000", 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7786000000000005e-05 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:133 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'91, mlcod=17'85)", 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.553679658, 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058524220000000004, 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.505 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918003+0000", 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918005+0000", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2899999999999999e-06 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918016+0000", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1636e-05 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918045+0000", 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8863000000000001e-05 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918058+0000", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.322e-05 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918155+0000", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6428999999999994e-05 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918185+0000", 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0219999999999999e-05 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923763+0000", 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055783969999999997 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923856+0000", 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2367000000000005e-05 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:134 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'92, mlcod=17'87)", 2024-09-11T05:51:12.509 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.552750879000001, 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050191519999999998, 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918932+0000", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.918934+0000", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9470000000000002e-06 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919003+0000", 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9258999999999995e-05 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919033+0000", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9884000000000001e-05 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919050+0000", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6503000000000001e-05 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919179+0000", 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012948199999999999 2024-09-11T05:51:12.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919236+0000", 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6348999999999999e-05 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923866+0000", 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046299920000000003 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923951+0000", 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5735999999999994e-05 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:135 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'93, mlcod=17'87)", 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.552236646000001, 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046106410000000004, 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.514 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919446+0000", 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919449+0000", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2170000000000001e-06 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919468+0000", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9490000000000001e-05 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919500+0000", 2024-09-11T05:51:12.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1469e-05 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.919514+0000", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.486e-05 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920637+0000", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011221149999999999 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920690+0000", 2024-09-11T05:51:12.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3328999999999997e-05 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923958+0000", 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032681020000000002 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924057+0000", 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9059000000000001e-05 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:136 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'94, mlcod=17'87)", 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.551323103000001, 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037872240000000001, 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920360+0000", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920362+0000", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.844e-06 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920374+0000", 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2636000000000001e-05 2024-09-11T05:51:12.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920403+0000", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9006e-05 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.920417+0000", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.331e-05 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921463+0000", 2024-09-11T05:51:12.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010457719999999999 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921495+0000", 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2642999999999999e-05 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924065+0000", 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025699239999999999 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924147+0000", 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2089000000000006e-05 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:137 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'95, mlcod=17'88)", 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.550102652, 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044436249999999997, 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.523 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921580+0000", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921582+0000", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.883e-06 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921642+0000", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9735999999999999e-05 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921675+0000", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2846999999999999e-05 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.921688+0000", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3390000000000001e-05 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922264+0000", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057572799999999996 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.922341+0000", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6779999999999993e-05 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925964+0000", 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003623458 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926024+0000", 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9803000000000003e-05 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:138 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'96, mlcod=17'88)", 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.548372491999999, 2024-09-11T05:51:12.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027531919999999998, 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923311+0000", 2024-09-11T05:51:12.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923312+0000", 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5030000000000001e-06 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923324+0000", 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1667e-05 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923347+0000", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3743e-05 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.923360+0000", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3013000000000001e-05 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924276+0000", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091579300000000003 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924332+0000", 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6073000000000003e-05 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926027+0000", 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00169487 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926064+0000", 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6529999999999998e-05 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:139 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'97, mlcod=17'89)", 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.547298887, 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038371870000000001, 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924384+0000", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924386+0000", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.683e-06 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924400+0000", 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4535999999999999e-05 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924426+0000", 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.533e-05 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.924439+0000", 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.343e-05 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925265+0000", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082617699999999999 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925354+0000", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8802999999999997e-05 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928126+0000", 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027723209999999999 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928221+0000", 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4907000000000005e-05 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:140 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'98, mlcod=17'93)", 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.545692973000001, 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049559219999999998, 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.536 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925990+0000", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.925992+0000", 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.533e-06 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926050+0000", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8875999999999997e-05 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926070+0000", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9259999999999999e-05 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926082+0000", 2024-09-11T05:51:12.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1960000000000001e-05 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926163+0000", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1493000000000001e-05 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.926192+0000", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8583000000000001e-05 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.930860+0000", 2024-09-11T05:51:12.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046681140000000001 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.930946+0000", 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6102999999999999e-05 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4176.0:141 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'99, mlcod=17'93)", 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.544265333999999, 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041853410000000004, 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:12.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927418+0000", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927419+0000", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927427+0000", 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-11T05:51:12.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927446+0000", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8683999999999999e-05 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.927455+0000", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928283+0000", 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082788000000000002 2024-09-11T05:51:12.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.928341+0000", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8183000000000002e-05 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.931544+0000", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032032250000000001 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:52.931603+0000", 2024-09-11T05:51:12.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8990000000000003e-05 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.9480561249999999, 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033108579999999999, 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:12.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523627+0000", 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523630+0000", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5969999999999999e-06 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523642+0000", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2802999999999999e-05 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523670+0000", 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.764e-05 2024-09-11T05:51:12.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523684+0000", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.431e-05 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.523789+0000", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010516100000000001 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.524003+0000", 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021329000000000001 2024-09-11T05:51:12.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.524306+0000", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030360000000000001 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.525515+0000", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012085799999999999 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526790+0000", 2024-09-11T05:51:12.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001275261 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526874+0000", 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3867000000000002e-05 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.526938+0000", 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3749000000000002e-05 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:13 3.12 3:48b35b7a:::rbd_object_map.11246a69a978:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.9423242580000002, 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029318399999999998, 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:12.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 13 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529359+0000", 2024-09-11T05:51:12.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529361+0000", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9460000000000001e-06 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529372+0000", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1623999999999999e-05 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529399+0000", 2024-09-11T05:51:12.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6352000000000001e-05 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529412+0000", 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3647e-05 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529518+0000", 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105463 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529744+0000", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022606500000000001 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.529997+0000", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025276399999999999 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.531513+0000", 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015163150000000001 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532144+0000", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063138500000000004 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532227+0000", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3035999999999996e-05 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.532291+0000", 2024-09-11T05:51:12.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3243000000000004e-05 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:884 3.12 3:48b35b7a:::rbd_object_map.11246a69a978:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:11.472103+0000", 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.99958053599999996, 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026146310000000001, 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:12.557 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 884 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472103+0000", 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472103+0000", 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472105+0000", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472118+0000", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3196e-05 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472141+0000", 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2853e-05 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472152+0000", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1231e-05 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472244+0000", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1172000000000001e-05 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472339+0000", 2024-09-11T05:51:12.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5904999999999997e-05 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.472653+0000", 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031401699999999999 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.473582+0000", 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00092847899999999996 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.474636+0000", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00105367 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.474678+0000", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2536000000000001e-05 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.474717+0000", 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8977999999999998e-05 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.562 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:912 3.12 3:48b35b7a:::rbd_object_map.11246a69a978:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:11.517851+0000", 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.95383180499999998, 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002719333, 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:12.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 912 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517851+0000", 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:12.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517851+0000", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517853+0000", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6109999999999999e-06 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517861+0000", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1319999999999994e-06 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:12.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517880+0000", 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9318e-05 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517891+0000", 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0278000000000001e-05 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.517958+0000", 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7426000000000001e-05 2024-09-11T05:51:12.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.518070+0000", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112419 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.518483+0000", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041232999999999998 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.519857+0000", 2024-09-11T05:51:12.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013739430000000001 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.520433+0000", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057636600000000003 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.520512+0000", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8719000000000003e-05 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.520571+0000", 2024-09-11T05:51:12.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8791e-05 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:12.569 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:12.569 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:12.470+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:12.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:12.611 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:12.610+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:12.675 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:12.674+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:12.744 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:12.744 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:12.744 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:12.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:12.777 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:12.778+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:12.837 INFO:teuthology.orchestra.run.smithi165.stdout:383 read 0x36bc669 thru 0x36c63e6 (0x9d7e bytes) 2024-09-11T05:51:12.838 INFO:teuthology.orchestra.run.smithi165.stdout:385 write 0x4bc118c thru 0x4bcccda (0xbb4f bytes) 2024-09-11T05:51:12.853 INFO:teuthology.orchestra.run.smithi165.stdout:386 read 0xafac443 thru 0xafb6225 (0x9de3 bytes) 2024-09-11T05:51:12.853 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:12.864 INFO:teuthology.orchestra.run.smithi165.stdout:389 writesame 0xec15f7a thru 0xec24479 data_size 0x80(0xe500 bytes) 2024-09-11T05:51:12.865 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:12.865 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:12.865 INFO:teuthology.orchestra.run.smithi165.stdout:395 write 0x904f92e thru 0x905ccf3 (0xd3c6 bytes) 2024-09-11T05:51:12.874 INFO:teuthology.orchestra.run.smithi165.stdout:397 read 0x3e2900b thru 0x3e35c97 (0xcc8d bytes) 2024-09-11T05:51:12.874 INFO:teuthology.orchestra.run.smithi165.stdout:399 read 0xc53b512 thru 0xc53bbb2 (0x6a1 bytes) 2024-09-11T05:51:12.874 INFO:teuthology.orchestra.run.smithi165.stdout:400 trunc from 0xec2447a to 0xe4abec9 2024-09-11T05:51:12.877 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:12.874+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:12.882 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:12.882 INFO:teuthology.orchestra.run.smithi165.stdout:402 read 0xa08c727 thru 0xa097509 (0xade3 bytes) 2024-09-11T05:51:12.882 INFO:teuthology.orchestra.run.smithi165.stdout:404 writesame 0x36dae5b thru 0x36dbbc9 data_size 0x1(0xd6f bytes) 2024-09-11T05:51:12.903 INFO:teuthology.orchestra.run.smithi165.stdout:406 trunc from 0xe4abec9 to 0x3eada72 2024-09-11T05:51:12.936 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:12.936 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:12.936 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:12.936 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:12.936 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:12.947 INFO:teuthology.orchestra.run.smithi165.stdout:407 write 0x2ac45ee thru 0x2ac5d55 (0x1768 bytes) 2024-09-11T05:51:12.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:12.954 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:12.954 INFO:teuthology.orchestra.run.smithi165.stdout:409 punch from 0x39e2a48 to 0x39e5a3b, (0x2ff3 bytes) 2024-09-11T05:51:12.955 INFO:teuthology.orchestra.run.smithi165.stdout:410 read 0x32b7bd8 thru 0x32c4e53 (0xd27c bytes) 2024-09-11T05:51:12.955 INFO:teuthology.orchestra.run.smithi165.stdout:411 punch from 0xff0aca to 0xffce25, (0xc35b bytes) 2024-09-11T05:51:12.955 INFO:teuthology.orchestra.run.smithi165.stdout:413 read 0x5d176f thru 0x5d47da (0x306c bytes) 2024-09-11T05:51:12.975 INFO:teuthology.orchestra.run.smithi165.stdout:414 writesame 0xbffc10e thru 0xc00aecc data_size 0xb(0xedbf bytes) 2024-09-11T05:51:12.975 INFO:teuthology.orchestra.run.smithi165.stdout:415 read 0x4b38a4 thru 0x4bb34e (0x7aab bytes) 2024-09-11T05:51:12.978 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:12.978+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:12.984 INFO:teuthology.orchestra.run.smithi165.stdout:416 write 0xd792957 thru 0xd7930c5 (0x76f bytes) 2024-09-11T05:51:12.991 INFO:teuthology.orchestra.run.smithi165.stdout:417 read 0x86ff81e thru 0x86ffdf6 (0x5d9 bytes) 2024-09-11T05:51:12.991 INFO:teuthology.orchestra.run.smithi165.stdout:418 punch from 0xd3d6dd to 0xd48cca, (0xb5ed bytes) 2024-09-11T05:51:12.991 INFO:teuthology.orchestra.run.smithi165.stdout:419 write 0x334a1ab thru 0x334d545 (0x339b bytes) 2024-09-11T05:51:13.018 INFO:teuthology.orchestra.run.smithi165.stdout:420 trunc from 0xd7930c6 to 0x53ad8c 2024-09-11T05:51:13.069 INFO:teuthology.orchestra.run.smithi165.stdout:421 writesame 0x3198f03 thru 0x31a74a3 data_size 0x5(0xe5a1 bytes) 2024-09-11T05:51:13.070 INFO:teuthology.orchestra.run.smithi165.stdout:422 write 0x382da thru 0x3f54a (0x7271 bytes) 2024-09-11T05:51:13.079 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:13.078+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:13.089 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:5 2.2 e21/19 2:5cea7035:::rbd_info:head v 21'1, mlcod=0'0)", 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.921571663, 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017785089999999999, 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166259+0000", 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166260+0000", 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6699999999999999e-07 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166268+0000", 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.276e-06 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166293+0000", 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.527e-05 2024-09-11T05:51:13.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166311+0000", 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7439e-05 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166434+0000", 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012318299999999999 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.166493+0000", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9083000000000003e-05 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.167936+0000", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014426759999999999 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:57.168038+0000", 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010211499999999999 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:7 2.2 e22/19 2:5cea7035:::rbd_info:head v 22'2, mlcod=0'0)", 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.9146052, 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019653880000000002, 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173226+0000", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173227+0000", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6599999999999996e-07 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173241+0000", 2024-09-11T05:51:13.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4243999999999999e-05 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173267+0000", 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6063000000000001e-05 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173297+0000", 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0513000000000001e-05 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173470+0000", 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017282500000000001 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.173553+0000", 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2637000000000002e-05 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.175108+0000", 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001555641 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:58.175191+0000", 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2799000000000005e-05 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:9 3.1a e26/23 3:5cea7035:::rbd_info:head v 26'2, mlcod=0'0)", 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5773475890000004, 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001489523, 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510484+0000", 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8899999999999999e-07 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510491+0000", 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1400000000000002e-06 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510515+0000", 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4102999999999999e-05 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510529+0000", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3208e-05 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510640+0000", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011099600000000001 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.510668+0000", 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8379999999999999e-05 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.511878+0000", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001209711 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:07.511973+0000", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5396000000000006e-05 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:953 3.19 3:9b18f687:::rbd_header.112475afb12f:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:13.103 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.47604059399999998, 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069395289999999998, 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 953 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611793+0000", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611810+0000", 2024-09-11T05:51:13.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6133999999999998e-05 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611844+0000", 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4403000000000002e-05 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611861+0000", 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7246999999999999e-05 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611991+0000", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012932899999999999 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.612003+0000", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2106000000000001e-05 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616192+0000", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041892049999999997 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616412+0000", 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021982100000000001 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616807+0000", 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039489700000000001 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618206+0000", 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013989510000000001 2024-09-11T05:51:13.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618641+0000", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043535200000000002 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618695+0000", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3740999999999999e-05 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618730+0000", 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:13.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:954 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.619025+0000", 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.46880646599999998, 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017969520000000001, 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 954 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619025+0000", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619025+0000", 2024-09-11T05:51:13.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619027+0000", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8540000000000001e-06 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619043+0000", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5569999999999998e-05 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619080+0000", 2024-09-11T05:51:13.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7126000000000003e-05 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619102+0000", 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619179+0000", 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6851e-05 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619250+0000", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0621e-05 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.619457+0000", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020699 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.620579+0000", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001122437 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.620683+0000", 2024-09-11T05:51:13.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010346800000000001 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.620759+0000", 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6524000000000001e-05 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.620822+0000", 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2374999999999998e-05 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:968 3.19 3:9b18f687:::rbd_header.112475afb12f:head [watch watch cookie 140146796586496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.639276+0000", 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.44855503899999999, 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018659379999999999, 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 968 2024-09-11T05:51:13.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639276+0000", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639276+0000", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639278+0000", 2024-09-11T05:51:13.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5460000000000001e-06 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639285+0000", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0319999999999998e-06 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639303+0000", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639315+0000", 2024-09-11T05:51:13.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2038000000000001e-05 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639385+0000", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9461000000000005e-05 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639493+0000", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108485 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.639724+0000", 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023049100000000001 2024-09-11T05:51:13.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.640770+0000", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001045902 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641037+0000", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000267163 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641082+0000", 2024-09-11T05:51:13.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5596999999999997e-05 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641142+0000", 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9549999999999997e-05 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:969 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.parent_attach in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.641316+0000", 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.4465152, 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001680045, 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.121 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 969 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641316+0000", 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641316+0000", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641317+0000", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641324+0000", 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5370000000000003e-06 2024-09-11T05:51:13.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641342+0000", 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8403999999999999e-05 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641352+0000", 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3819999999999993e-06 2024-09-11T05:51:13.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641403+0000", 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0945000000000001e-05 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641471+0000", 2024-09-11T05:51:13.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8285000000000001e-05 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.641676+0000", 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020483400000000001 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.642310+0000", 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063438399999999999 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.642922+0000", 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061225699999999995 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.642963+0000", 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1035000000000003e-05 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.642996+0000", 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2456000000000002e-05 2024-09-11T05:51:13.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:978 3.19 3:9b18f687:::rbd_header.112475afb12f:head [watch unwatch cookie 140146796586496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.654314+0000", 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43351715600000001, 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019718700000000001, 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 978 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654314+0000", 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654314+0000", 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654316+0000", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.835e-06 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654327+0000", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0868e-05 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654352+0000", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4871999999999998e-05 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654367+0000", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5313999999999999e-05 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654482+0000", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115654 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654633+0000", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015043899999999999 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.654890+0000", 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000257175 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.655816+0000", 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00092583300000000004 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.656182+0000", 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036602899999999997 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.656226+0000", 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4453000000000003e-05 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.656286+0000", 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9398000000000003e-05 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.133 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1015 3.19 3:9b18f687:::rbd_header.112475afb12f:head [watch watch cookie 140146594945760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.837539+0000", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.25029191899999997, 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017896520000000001, 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1015 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837539+0000", 2024-09-11T05:51:13.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837539+0000", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837542+0000", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837552+0000", 2024-09-11T05:51:13.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0105999999999999e-05 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837581+0000", 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.957e-05 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837603+0000", 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1946e-05 2024-09-11T05:51:13.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837675+0000", 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1347000000000007e-05 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837726+0000", 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1152000000000001e-05 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.837837+0000", 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111043 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.838547+0000", 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071038099999999995 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.839254+0000", 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070711799999999996 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.839286+0000", 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1461000000000003e-05 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.839329+0000", 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3154000000000001e-05 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1029 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.862573+0000", 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.225258404, 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001387714, 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1029 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862573+0000", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862573+0000", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862574+0000", 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:51:13.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862583+0000", 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4020000000000002e-06 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862601+0000", 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8054e-05 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862608+0000", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2989999999999999e-06 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862662+0000", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3514999999999998e-05 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862717+0000", 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5464999999999998e-05 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.862846+0000", 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128904 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.863366+0000", 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051977800000000004 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.863886+0000", 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000520402 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.863919+0000", 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2373000000000003e-05 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.863960+0000", 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1782000000000001e-05 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1041 3.0 3:008fdb6a:::rbd_data.112475afb12f.0000000000000091:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write 32768~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.871437+0000", 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.21639446400000001, 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001735917, 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1041 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871437+0000", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871437+0000", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871438+0000", 2024-09-11T05:51:13.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1400000000000001e-06 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871468+0000", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0550999999999999e-05 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871482+0000", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4138000000000001e-05 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871491+0000", 2024-09-11T05:51:13.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0539999999999995e-06 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871549+0000", 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7991000000000001e-05 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871642+0000", 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3644e-05 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:13.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871757+0000", 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114782 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872624+0000", 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086724600000000001 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.873109+0000", 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048488099999999999 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.873145+0000", 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6245000000000003e-05 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.873173+0000", 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7245000000000001e-05 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.151 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1043 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.877486+0000", 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.21034496699999999, 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001464822, 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1043 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877486+0000", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877486+0000", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877489+0000", 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.92e-06 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877505+0000", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6296000000000001e-05 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877539+0000", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3513999999999998e-05 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877562+0000", 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3260000000000001e-05 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877650+0000", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7836000000000004e-05 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877719+0000", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9370999999999997e-05 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.877881+0000", 2024-09-11T05:51:13.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161254 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.878448+0000", 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056705099999999995 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.878883+0000", 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043502800000000001 2024-09-11T05:51:13.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.878917+0000", 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4422999999999999e-05 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.878951+0000", 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3868999999999997e-05 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.157 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1077 3.4 3:263082b7:::rbd_data.112475afb12f.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,truncate 983040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.939477+0000", 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.148353973, 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017094580000000001, 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1077 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.158 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939477+0000", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939477+0000", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939480+0000", 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3100000000000001e-06 2024-09-11T05:51:13.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939495+0000", 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4663e-05 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939528+0000", 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3213999999999997e-05 2024-09-11T05:51:13.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939544+0000", 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5553000000000002e-05 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939620+0000", 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6501999999999994e-05 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939758+0000", 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137952 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939950+0000", 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000192039 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940855+0000", 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00090459799999999995 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.941122+0000", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000267011 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.941160+0000", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8222000000000001e-05 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.941187+0000", 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6393999999999998e-05 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.163 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1090 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.972618+0000", 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.115213541, 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001403074, 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.164 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1090 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972618+0000", 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972618+0000", 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972619+0000", 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6530000000000001e-06 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972628+0000", 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9630000000000004e-06 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972649+0000", 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0469999999999999e-05 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972660+0000", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1452e-05 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972727+0000", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6921000000000005e-05 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972788+0000", 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1393e-05 2024-09-11T05:51:13.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.972935+0000", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146506 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.973543+0000", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060783600000000005 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.973960+0000", 2024-09-11T05:51:13.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041696099999999998 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.973993+0000", 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3275999999999998e-05 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.974021+0000", 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7642999999999999e-05 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1099 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.982380+0000", 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.10545131100000001, 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014052789999999999, 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1099 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982380+0000", 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982380+0000", 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982382+0000", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8500000000000001e-06 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982391+0000", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.115e-06 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982411+0000", 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0163e-05 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982422+0000", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.133e-05 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982486+0000", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3699000000000001e-05 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982554+0000", 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8293000000000005e-05 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.982693+0000", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013836400000000001 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.983401+0000", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070815299999999999 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.983723+0000", 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032239200000000001 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.983756+0000", 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2369999999999997e-05 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.983785+0000", 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.955e-05 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1122 3.4 3:263082b7:::rbd_data.112475afb12f.000000000000003e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.027119+0000", 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.060712416999999998, 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031691110000000001, 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1122 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027119+0000", 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027119+0000", 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027121+0000", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0150000000000002e-06 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027129+0000", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6549999999999993e-06 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027153+0000", 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3442999999999999e-05 2024-09-11T05:51:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027165+0000", 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1741e-05 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027249+0000", 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4456999999999995e-05 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027310+0000", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-05 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027390+0000", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0345000000000004e-05 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.028603+0000", 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012125549999999999 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030194+0000", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001590883 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030236+0000", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2261000000000001e-05 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030288+0000", 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1876000000000003e-05 2024-09-11T05:51:13.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1138 3.4 3:24dd5b71:::rbd_data.112475afb12f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,truncate 622592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.043921+0000", 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.04391018, 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001676252, 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.182 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1138 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043921+0000", 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043921+0000", 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043923+0000", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7910000000000001e-06 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043931+0000", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9549999999999992e-06 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043949+0000", 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8771999999999999e-05 2024-09-11T05:51:13.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043961+0000", 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1990999999999999e-05 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044031+0000", 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9757000000000003e-05 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044203+0000", 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000171402 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044409+0000", 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020673100000000001 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045024+0000", 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061465399999999996 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045510+0000", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048636799999999999 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045559+0000", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8594999999999998e-05 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045597+0000", 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8235999999999998e-05 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1140 3.19 3:9c3d53d4:::rbd_data.112475afb12f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,truncate 589824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.045143+0000", 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.042687828999999997, 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001633539, 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1140 2024-09-11T05:51:13.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045143+0000", 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045143+0000", 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045144+0000", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.074e-06 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045152+0000", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3939999999999998e-06 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045166+0000", 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.451e-05 2024-09-11T05:51:13.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045176+0000", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2879999999999998e-06 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045247+0000", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1657999999999997e-05 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045405+0000", 2024-09-11T05:51:13.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015782799999999999 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045608+0000", 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020312600000000001 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046311+0000", 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070245099999999999 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046655+0000", 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034422400000000002 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046729+0000", 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4012999999999998e-05 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046777+0000", 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7973000000000003e-05 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.193 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1143 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.051472+0000", 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.036358676999999999, 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001406245, 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.194 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1143 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051472+0000", 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051472+0000", 2024-09-11T05:51:13.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051475+0000", 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3319999999999999e-06 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051487+0000", 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2522e-05 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051514+0000", 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6342999999999998e-05 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051527+0000", 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3437e-05 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051595+0000", 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7960999999999995e-05 2024-09-11T05:51:13.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051666+0000", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1151999999999999e-05 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.051861+0000", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019514200000000001 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.052395+0000", 2024-09-11T05:51:13.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053355700000000004 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.052807+0000", 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041173499999999999 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.052846+0000", 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.977e-05 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.052879+0000", 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2293999999999999e-05 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1153 3.19 3:9b18f687:::rbd_header.112475afb12f:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.066851+0000", 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.020979662999999999, 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015915969999999999, 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1153 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066851+0000", 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066851+0000", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066854+0000", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.244e-06 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066864+0000", 2024-09-11T05:51:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0362e-05 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066889+0000", 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5177999999999998e-05 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066901+0000", 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1892000000000001e-05 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.066985+0000", 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4241999999999997e-05 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.067064+0000", 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9098999999999996e-05 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.067230+0000", 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016580300000000001 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.067778+0000", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054820100000000001 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.068371+0000", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00059209200000000001 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.068411+0000", 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0014e-05 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.068443+0000", 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2469999999999999e-05 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.206 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:13.207 INFO:teuthology.orchestra.run.smithi165.stdout:423 write 0xc13fb64 thru 0xc146251 (0x66ee bytes) 2024-09-11T05:51:13.207 INFO:teuthology.orchestra.run.smithi165.stdout:426 write 0x7d3736c thru 0x7d3c555 (0x51ea bytes) 2024-09-11T05:51:13.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:427 punch from 0x3fb7c8a to 0x3fc374b, (0xbac1 bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:429 writesame 0x8246213 thru 0x825607f data_size 0x3(0xfe6d bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:431 read 0x6d0468b thru 0x6d0f212 (0xab88 bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:434 read 0x12be97c thru 0x12c6eae (0x8533 bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:436 write 0xa56dc86 thru 0xa57c5b7 (0xe932 bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:438 write 0x7e19f2 thru 0x7e83ea (0x69f9 bytes) 2024-09-11T05:51:13.209 INFO:teuthology.orchestra.run.smithi165.stdout:439 punch from 0x15f922f to 0x15fe149, (0x4f1a bytes) 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:443 write 0x56a21b9 thru 0x56ab828 (0x9670 bytes) 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:445 writesame 0xb7ba768 thru 0xb7bf876 data_size 0x1(0x510f bytes) 2024-09-11T05:51:13.210 INFO:teuthology.orchestra.run.smithi165.stdout:446 trunc from 0xc146252 to 0x2578889 2024-09-11T05:51:13.210 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:13.206+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:13.214 INFO:teuthology.orchestra.run.smithi165.stdout:447 read 0x242caf7 thru 0x242e099 (0x15a3 bytes) 2024-09-11T05:51:13.225 INFO:teuthology.orchestra.run.smithi165.stdout:448 trunc from 0x2578889 to 0x705543c 2024-09-11T05:51:13.263 INFO:teuthology.orchestra.run.smithi165.stdout:450 writesame 0xeb24678 thru 0xeb30398 data_size 0x1(0xbd21 bytes) 2024-09-11T05:51:13.263 INFO:teuthology.orchestra.run.smithi165.stdout:451 read 0x604766a thru 0x6055a8c (0xe423 bytes) 2024-09-11T05:51:13.263 INFO:teuthology.orchestra.run.smithi165.stdout:454 write 0x436be48 thru 0x437258b (0x6744 bytes) 2024-09-11T05:51:13.291 INFO:teuthology.orchestra.run.smithi165.stdout:455 trunc from 0xeb30399 to 0x11e8edd 2024-09-11T05:51:13.308 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:13.306+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:13.310 INFO:teuthology.orchestra.run.smithi165.stdout:456 clone 2 order 22 su 16384 sc 6 2024-09-11T05:51:13.348 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:13.348 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:13.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:13.408 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:13.410+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:13.509 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:13.506+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:13.527 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:13.527 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:13.527 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:13.527 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:13.527 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:13.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:13.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:51:13.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:13.631 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:13.630+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:13.646 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone1 from 0x11e8edd (overlap 0x53ad8c) to 0x29d6c6 2024-09-11T05:51:13.712 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:13.714+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:13.730 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:13.730 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:13.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:13.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:1 2.3 e20/19 2:c4c92e5a:::rbd_trash:head v 20'1, mlcod=0'0)", 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.572747544999999, 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017074670000000001, 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:13.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158545+0000", 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4e-07 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158565+0000", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0299999999999999e-05 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158591+0000", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6146e-05 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158613+0000", 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1647000000000001e-05 2024-09-11T05:51:13.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158753+0000", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014034200000000001 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158836+0000", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3249000000000002e-05 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160169+0000", 2024-09-11T05:51:13.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013321769999999999 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160251+0000", 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2766e-05 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1041 3.0 e30/23 3:008fdb6a:::rbd_data.112475afb12f.0000000000000091:head v 30'3, mlcod=28'1)", 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.871947+0000", 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.85934417699999999, 2024-09-11T05:51:13.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000953271, 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871947+0000", 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871947+0000", 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871947+0000", 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.871986+0000", 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8334999999999998e-05 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872000+0000", 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3871e-05 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872010+0000", 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0399e-05 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872074+0000", 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3881999999999996e-05 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872093+0000", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8933999999999999e-05 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872863+0000", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077038599999999997 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.872900+0000", 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7248e-05 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1058 3.0 e30/23 3:008fdb6a:::rbd_data.112475afb12f.0000000000000091:head v 30'4, mlcod=28'2)", 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.909541+0000", 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.82175007799999999, 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072309999999999996, 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909541+0000", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909541+0000", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909543+0000", 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7230000000000001e-06 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909554+0000", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866999999999999e-05 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909574+0000", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9808000000000001e-05 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909585+0000", 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0995000000000001e-05 2024-09-11T05:51:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909689+0000", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104363 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.909713+0000", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3751e-05 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.910228+0000", 2024-09-11T05:51:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051483300000000002 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.910264+0000", 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.676e-05 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1077 3.4 e30/23 3:263082b7:::rbd_data.112475afb12f.000000000000003e:head v 30'13, mlcod=28'11)", 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.940105+0000", 2024-09-11T05:51:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.79118684399999994, 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086285099999999998, 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940105+0000", 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940105+0000", 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940106+0000", 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.84e-06 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940119+0000", 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2999999999999999e-05 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940145+0000", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5494999999999999e-05 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940160+0000", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5157000000000001e-05 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940226+0000", 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5696000000000003e-05 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940243+0000", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7331000000000001e-05 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940928+0000", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068513500000000002 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.940967+0000", 2024-09-11T05:51:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9196999999999998e-05 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1122 3.4 e30/23 3:263082b7:::rbd_data.112475afb12f.000000000000003e:head v 30'14, mlcod=28'12)", 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.027663+0000", 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.703628268, 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080584199999999997, 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027663+0000", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027663+0000", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027665+0000", 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.708e-06 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027676+0000", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1015e-05 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027696+0000", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0231e-05 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027707+0000", 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0777e-05 2024-09-11T05:51:13.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027773+0000", 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6400000000000001e-05 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027814+0000", 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0534000000000002e-05 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.028423+0000", 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060891099999999996 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.028469+0000", 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6266000000000001e-05 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1138 3.4 e30/23 3:24dd5b71:::rbd_data.112475afb12f.0000000000000004:head v 30'15, mlcod=30'13)", 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.044508+0000", 2024-09-11T05:51:13.753 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.68678348199999995, 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000836624, 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044508+0000", 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044508+0000", 2024-09-11T05:51:13.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044509+0000", 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.232e-06 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044517+0000", 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3489999999999992e-06 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044535+0000", 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7983999999999999e-05 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044543+0000", 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5109999999999997e-06 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044606+0000", 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489000000000001e-05 2024-09-11T05:51:13.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044631+0000", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4643000000000001e-05 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045304+0000", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067322200000000001 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045345+0000", 2024-09-11T05:51:13.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0194000000000002e-05 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1174 3.4 e30/23 3:2099f16f:::rbd_data.112475afb12f.00000000000000be:head v 30'16, mlcod=30'14)", 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.100158+0000", 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.63113375900000002, 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011454690000000001, 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100158+0000", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100158+0000", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100159+0000", 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6980000000000001e-06 2024-09-11T05:51:13.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100181+0000", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1319999999999999e-05 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100204+0000", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3405e-05 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100215+0000", 2024-09-11T05:51:13.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1058e-05 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100293+0000", 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8264999999999997e-05 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.100324+0000", 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0332999999999999e-05 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.101243+0000", 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091912700000000005 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.101303+0000", 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0263000000000002e-05 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1212 3.4 e30/23 3:2099f16f:::rbd_data.112475afb12f.00000000000000be:head v 30'17, mlcod=30'15)", 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:13.762 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.53524168800000005, 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001644195, 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:13.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196052+0000", 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7939999999999998e-06 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196072+0000", 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9259e-05 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196121+0000", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-05 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196164+0000", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3016999999999998e-05 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196311+0000", 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147188 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196367+0000", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6597e-05 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.197580+0000", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001212273 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.197694+0000", 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114207 2024-09-11T05:51:13.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:13.767 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-09-11T05:51:13.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:13.769 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:13.769 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:13.776 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:13.776 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:13.810 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:13.810 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:13.813 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:13.810+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:13.840 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:13.840 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:13.861 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:13.861 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:13.894 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:13.894 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:13.908 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:13.908 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:13.914 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:13.910+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:13.916 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:13.916 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:13.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:13.937 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:13.938 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:14.016 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:14.014+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:14.056 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.056 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:14.116 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:14.114+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:14.123 INFO:teuthology.orchestra.run.smithi165.stdout:457 writesame 0x1efd686 thru 0x1f09c8d data_size 0x4(0xc608 bytes) 2024-09-11T05:51:14.141 INFO:teuthology.orchestra.run.smithi165.stdout:459 write 0x6e19f31 thru 0x6e26283 (0xc353 bytes) 2024-09-11T05:51:14.160 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:14.160 INFO:teuthology.orchestra.run.smithi165.stdout:462 read 0x3a3a923 thru 0x3a3f98c (0x506a bytes) 2024-09-11T05:51:14.160 INFO:teuthology.orchestra.run.smithi165.stdout:463 read 0x30469bb thru 0x304adc1 (0x4407 bytes) 2024-09-11T05:51:14.169 INFO:teuthology.orchestra.run.smithi165.stdout:464 write 0x7943a0b thru 0x794e181 (0xa777 bytes) 2024-09-11T05:51:14.186 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:14.186 INFO:teuthology.orchestra.run.smithi165.stdout:466 write 0x33aef8 thru 0x33af88 (0x91 bytes) 2024-09-11T05:51:14.216 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:14.214+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:14.227 INFO:teuthology.orchestra.run.smithi165.stdout:468 write 0x9111377 thru 0x911b178 (0x9e02 bytes) 2024-09-11T05:51:14.239 INFO:teuthology.orchestra.run.smithi165.stdout:472 punch from 0x9d287f to 0x9d5327, (0x2aa8 bytes) 2024-09-11T05:51:14.240 INFO:teuthology.orchestra.run.smithi165.stdout:474 write 0x54b6be thru 0x5500a2 (0x49e5 bytes) 2024-09-11T05:51:14.242 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.242 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:14.291 INFO:teuthology.orchestra.run.smithi165.stdout:475 write 0xd69e15c thru 0xd6ad521 (0xf3c6 bytes) 2024-09-11T05:51:14.303 INFO:teuthology.orchestra.run.smithi165.stdout:478 read 0x38dd453 thru 0x38e8adc (0xb68a bytes) 2024-09-11T05:51:14.317 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:14.318+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:14.325 INFO:teuthology.orchestra.run.smithi165.stdout:479 trunc from 0xd6ad522 to 0x90d0eb 2024-09-11T05:51:14.377 INFO:teuthology.orchestra.run.smithi165.stdout:480 read 0x787996 thru 0x7937a4 (0xbe0f bytes) 2024-09-11T05:51:14.378 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:14.378 INFO:teuthology.orchestra.run.smithi165.stdout:482 clone 3 order 20 su 4096 sc 3 2024-09-11T05:51:14.404 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.404 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:14.419 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:14.418+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:14.519 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:14.518+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:14.551 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:51:14.584 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.584 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:14.644 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:14.646+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:14.666 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone2 from 0x90d0eb (overlap 0x90d0eb) to 0x655f0a 2024-09-11T05:51:14.720 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:14.718+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:14.729 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.729 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:14.821 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:14.818+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:14.834 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-09-11T05:51:14.888 INFO:teuthology.orchestra.run.smithi165.stdout:483 writesame 0x5ccf70a thru 0x5cdef17 data_size 0x1(0xf80e bytes) 2024-09-11T05:51:14.908 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:14.909 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:14.916 INFO:teuthology.orchestra.run.smithi165.stdout:484 writesame 0xaaed117 thru 0xaaf6c0f data_size 0x1(0x9af9 bytes) 2024-09-11T05:51:14.916 INFO:teuthology.orchestra.run.smithi165.stdout:485 read 0x5d99463 thru 0x5da1839 (0x83d7 bytes) 2024-09-11T05:51:14.916 INFO:teuthology.orchestra.run.smithi165.stdout:486 read 0x2316b thru 0x24691 (0x1527 bytes) 2024-09-11T05:51:14.922 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:14.918+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:14.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:14.947 INFO:teuthology.orchestra.run.smithi165.stdout:487 trunc from 0xaaf6c10 to 0x2f3117 2024-09-11T05:51:15.023 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:15.022+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:15.057 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.057 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:15.080 INFO:teuthology.orchestra.run.smithi165.stdout:488 punch from 0x2d3aae to 0x2d3c4f, (0x1a1 bytes) 2024-09-11T05:51:15.080 INFO:teuthology.orchestra.run.smithi165.stdout:489 read 0x1658b3 thru 0x16cc99 (0x73e7 bytes) 2024-09-11T05:51:15.088 INFO:teuthology.orchestra.run.smithi165.stdout:492 trunc from 0x2f3117 to 0x3445469 2024-09-11T05:51:15.097 INFO:teuthology.orchestra.run.smithi165.stdout:495 punch from 0x309bbe3 to 0x30a9142, (0xd55f bytes) 2024-09-11T05:51:15.114 INFO:teuthology.orchestra.run.smithi165.stdout:497 writesame 0x6c1542d thru 0x6c19661 data_size 0x1(0x4235 bytes) 2024-09-11T05:51:15.114 INFO:teuthology.orchestra.run.smithi165.stdout:498 write 0x2059ba4 thru 0x205b7cf (0x1c2c bytes) 2024-09-11T05:51:15.123 INFO:teuthology.orchestra.run.smithi165.stdout:499 read 0x2c61cb5 thru 0x2c6d712 (0xba5e bytes) 2024-09-11T05:51:15.136 INFO:teuthology.orchestra.run.smithi165.stdout:500 trunc from 0x6c19662 to 0xc319574 2024-09-11T05:51:15.160 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:15.162+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:15.166 INFO:teuthology.orchestra.run.smithi165.stdout:502 write 0xede723c thru 0xedece02 (0x5bc7 bytes) 2024-09-11T05:51:15.194 INFO:teuthology.orchestra.run.smithi165.stdout:504 trunc from 0xedece03 to 0x79a691d 2024-09-11T05:51:15.219 INFO:teuthology.orchestra.run.smithi165.stdout:506 trunc from 0x79a691d to 0x68774b7 2024-09-11T05:51:15.225 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:15.226+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:15.233 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.243 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.243 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.250 INFO:teuthology.orchestra.run.smithi165.stdout:510 trunc from 0x68774b7 to 0xd9fe9e1 2024-09-11T05:51:15.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:511 punch from 0x2b80471 to 0x2b82f35, (0x2ac4 bytes) 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:514 read 0x9bf3270 thru 0x9bf9412 (0x61a3 bytes) 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:515 read 0x40fa0b8 thru 0x4105b73 (0xbabc bytes) 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:516 punch from 0xa033202 to 0xa042a4e, (0xf84c bytes) 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:517 punch from 0xd2fdc52 to 0xd305c97, (0x8045 bytes) 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.264 INFO:teuthology.orchestra.run.smithi165.stdout:519 write 0xc0c3fe3 thru 0xc0cb65d (0x767b bytes) 2024-09-11T05:51:15.274 INFO:teuthology.orchestra.run.smithi165.stdout:522 read 0x1218002 thru 0x121fd3c (0x7d3b bytes) 2024-09-11T05:51:15.274 INFO:teuthology.orchestra.run.smithi165.stdout:523 write 0x854d6a thru 0x86117b (0xc412 bytes) 2024-09-11T05:51:15.282 INFO:teuthology.orchestra.run.smithi165.stdout:524 punch from 0x5e8a0ad to 0x5e951ce, (0xb121 bytes) 2024-09-11T05:51:15.284 INFO:teuthology.orchestra.run.smithi165.stdout:525 read 0x2b12d1c thru 0x2b18789 (0x5a6e bytes) 2024-09-11T05:51:15.284 INFO:teuthology.orchestra.run.smithi165.stdout:528 read 0xcaf8fb4 thru 0xcb02ffc (0xa049 bytes) 2024-09-11T05:51:15.284 INFO:teuthology.orchestra.run.smithi165.stdout:529 write 0x2051abb thru 0x2056243 (0x4789 bytes) 2024-09-11T05:51:15.285 INFO:teuthology.orchestra.run.smithi165.stdout:530 write 0x96558e6 thru 0x965cca3 (0x73be bytes) 2024-09-11T05:51:15.297 INFO:teuthology.orchestra.run.smithi165.stdout:531 read 0x5dc3454 thru 0x5dcde9e (0xaa4b bytes) 2024-09-11T05:51:15.297 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.297 INFO:teuthology.orchestra.run.smithi165.stdout:533 write 0x4056684 thru 0x40600dd (0x9a5a bytes) 2024-09-11T05:51:15.308 INFO:teuthology.orchestra.run.smithi165.stdout:534 writesame 0x57f9d6f thru 0x5803c69 data_size 0x1(0x9efb bytes) 2024-09-11T05:51:15.325 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:15.322+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:15.341 INFO:teuthology.orchestra.run.smithi165.stdout:535 write 0xb9c1b88 thru 0xb9c2630 (0xaa9 bytes) 2024-09-11T05:51:15.348 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.348 INFO:teuthology.orchestra.run.smithi165.stdout:539 punch from 0xd3a989d to 0xd3ae54b, (0x4cae bytes) 2024-09-11T05:51:15.348 INFO:teuthology.orchestra.run.smithi165.stdout:541 writesame 0x55c9a5e thru 0x55d5e59 data_size 0x4a(0xc3fc bytes) 2024-09-11T05:51:15.360 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.360 INFO:teuthology.orchestra.run.smithi165.stdout:544 punch from 0x2ab7e70 to 0x2ac3b2d, (0xbcbd bytes) 2024-09-11T05:51:15.361 INFO:teuthology.orchestra.run.smithi165.stdout:545 read 0xb6e20c9 thru 0xb6f13e6 (0xf31e bytes) 2024-09-11T05:51:15.361 INFO:teuthology.orchestra.run.smithi165.stdout:547 read 0x99e2341 thru 0x99e77f4 (0x54b4 bytes) 2024-09-11T05:51:15.361 INFO:teuthology.orchestra.run.smithi165.stdout:549 read 0x92bbcf thru 0x92fade (0x3f10 bytes) 2024-09-11T05:51:15.361 INFO:teuthology.orchestra.run.smithi165.stdout:550 read 0x56fa87 thru 0x571b5c (0x20d6 bytes) 2024-09-11T05:51:15.361 INFO:teuthology.orchestra.run.smithi165.stdout:552 writesame 0xa36d0a thru 0xa3bcfa data_size 0x1(0x4ff1 bytes) 2024-09-11T05:51:15.382 INFO:teuthology.orchestra.run.smithi165.stdout:553 writesame 0x49d7a7d thru 0x49e6c44 data_size 0xc(0xf1c8 bytes) 2024-09-11T05:51:15.397 INFO:teuthology.orchestra.run.smithi165.stdout:554 read 0x92459fd thru 0x9254083 (0xe687 bytes) 2024-09-11T05:51:15.397 INFO:teuthology.orchestra.run.smithi165.stdout:555 read 0xcb1d558 thru 0xcb1f8da (0x2383 bytes) 2024-09-11T05:51:15.397 INFO:teuthology.orchestra.run.smithi165.stdout:558 read 0x88967b3 thru 0x8897d87 (0x15d5 bytes) 2024-09-11T05:51:15.420 INFO:teuthology.orchestra.run.smithi165.stdout:560 trunc from 0xd9fe9e1 to 0x4cfc8c7 2024-09-11T05:51:15.428 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:15.426+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:15.428 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.428 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:15.451 INFO:teuthology.orchestra.run.smithi165.stdout:561 clone 4 order 22 su 16384 sc 10 2024-09-11T05:51:15.526 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:15.522+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:15.551 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:15.551 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:15.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:15.570 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.570 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:15.636 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:15.638+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:15.641 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone3 from 0x4cfc8c7 (overlap 0x2f3117) to 0x1c50a7 2024-09-11T05:51:15.716 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.717 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.728 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:15.730+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:15.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:15.757 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-09-11T05:51:15.797 INFO:teuthology.orchestra.run.smithi165.stdout:563 writesame 0xb7225c7 thru 0xb722b70 data_size 0x19(0x5aa bytes) 2024-09-11T05:51:15.817 INFO:teuthology.orchestra.run.smithi165.stdout:564 trunc from 0xb722b71 to 0xb06423 2024-09-11T05:51:15.828 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:15.830+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:15.929 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:15.930+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:15.932 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:15.933 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:15.933 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:15.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:15.952 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-11T05:51:15.495535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13953522,"num_objects":48,"num_object_clones":13,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":4553,"num_read_kb":8848,"num_write":1105,"num_write_kb":9481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242444,"kb_used_data":9500,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732276,"statfs":{"total":773094113280,"available":772845850624,"internally_reserved":0,"allocated":9728000,"data_stored":8996898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8879168,"num_objects":30,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":3818,"num_read_kb":8226,"num_write":819,"num_write_kb":6494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.975238"},"pg_stats":[{"pgid":"3.1f","version":"31'89","reported_seq":202,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.832862+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:51:14.832862+0000","last_peered":"2024-09-11T05:51:14.832862+0000","last_clean":"2024-09-11T05:51:14.832862+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:51:14.832862+0000","last_undegraded":"2024-09-11T05:51:14.832862+0000","last_fullsized":"2024-09-11T05:51:14.832862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:22.505074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":589924,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":489,"num_read_kb":936,"num_write":146,"num_write_kb":212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"30'7","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:13.543907+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:51:13.543907+0000","last_peered":"2024-09-11T05:51:13.543907+0000","last_clean":"2024-09-11T05:51:13.543907+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:51:13.543907+0000","last_undegraded":"2024-09-11T05:51:13.543907+0000","last_fullsized":"2024-09-11T05:51:13.543907+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:27:24.399794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"31'2","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.584554+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:51:14.584554+0000","last_peered":"2024-09-11T05:51:14.584554+0000","last_clean":"2024-09-11T05:51:14.584554+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:51:14.584554+0000","last_undegraded":"2024-09-11T05:51:14.584554+0000","last_fullsized":"2024-09-11T05:51:14.584554+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:23:12.129626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"32'18","reported_seq":68,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.783138+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:51:14.783138+0000","last_peered":"2024-09-11T05:51:14.783138+0000","last_clean":"2024-09-11T05:51:14.783138+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:51:14.783138+0000","last_undegraded":"2024-09-11T05:51:14.783138+0000","last_fullsized":"2024-09-11T05:51:14.783138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:58:05.994336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":24,"num_write":18,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"30'4","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.565682+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:51:14.565682+0000","last_peered":"2024-09-11T05:51:14.565682+0000","last_clean":"2024-09-11T05:51:14.565682+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:51:14.565682+0000","last_undegraded":"2024-09-11T05:51:14.565682+0000","last_fullsized":"2024-09-11T05:51:14.565682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:58:35.254031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"31'7","reported_seq":227,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.831272+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:51:14.831272+0000","last_peered":"2024-09-11T05:51:14.831272+0000","last_clean":"2024-09-11T05:51:14.831272+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:51:14.831272+0000","last_undegraded":"2024-09-11T05:51:14.831272+0000","last_fullsized":"2024-09-11T05:51:14.831272+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:12:14.941337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":601503,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":710,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"32'44","reported_seq":318,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.840279+0000","last_change":"2024-09-11T05:51:07.514037+0000","last_active":"2024-09-11T05:51:14.840279+0000","last_peered":"2024-09-11T05:51:14.840279+0000","last_clean":"2024-09-11T05:51:14.840279+0000","last_became_active":"2024-09-11T05:51:05.507475+0000","last_became_peered":"2024-09-11T05:51:05.507475+0000","last_unstale":"2024-09-11T05:51:14.840279+0000","last_undegraded":"2024-09-11T05:51:14.840279+0000","last_fullsized":"2024-09-11T05:51:14.840279+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:13:57.403839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":589824,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":601,"num_read_kb":1032,"num_write":41,"num_write_kb":182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"31'24","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.123960+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:51:14.123960+0000","last_peered":"2024-09-11T05:51:14.123960+0000","last_clean":"2024-09-11T05:51:14.123960+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:51:14.123960+0000","last_undegraded":"2024-09-11T05:51:14.123960+0000","last_fullsized":"2024-09-11T05:51:14.123960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:12:13.967183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":66,"num_write":22,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"30'6","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.565756+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:51:14.565756+0000","last_peered":"2024-09-11T05:51:14.565756+0000","last_clean":"2024-09-11T05:51:14.565756+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:51:14.565756+0000","last_undegraded":"2024-09-11T05:51:14.565756+0000","last_fullsized":"2024-09-11T05:51:14.565756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:02:11.221506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"28'3","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.584397+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:51:14.584397+0000","last_peered":"2024-09-11T05:51:14.584397+0000","last_clean":"2024-09-11T05:51:14.584397+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:51:14.584397+0000","last_undegraded":"2024-09-11T05:51:14.584397+0000","last_fullsized":"2024-09-11T05:51:14.584397+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:31.928746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":3,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"30'10","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.574905+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:51:14.574905+0000","last_peered":"2024-09-11T05:51:14.574905+0000","last_clean":"2024-09-11T05:51:14.574905+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:51:14.574905+0000","last_undegraded":"2024-09-11T05:51:14.574905+0000","last_fullsized":"2024-09-11T05:51:14.574905+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:39.589522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"32'56","reported_seq":220,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.841016+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:51:14.841016+0000","last_peered":"2024-09-11T05:51:14.841016+0000","last_clean":"2024-09-11T05:51:14.841016+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:51:14.841016+0000","last_undegraded":"2024-09-11T05:51:14.841016+0000","last_fullsized":"2024-09-11T05:51:14.841016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:18.354900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":3118330,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":349,"num_read_kb":1690,"num_write":93,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575781+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:51:14.575781+0000","last_peered":"2024-09-11T05:51:14.575781+0000","last_clean":"2024-09-11T05:51:14.575781+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:51:14.575781+0000","last_undegraded":"2024-09-11T05:51:14.575781+0000","last_fullsized":"2024-09-11T05:51:14.575781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"32'28","reported_seq":62,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.565539+0000","last_change":"2024-09-11T05:51:07.518766+0000","last_active":"2024-09-11T05:51:14.565539+0000","last_peered":"2024-09-11T05:51:14.565539+0000","last_clean":"2024-09-11T05:51:14.565539+0000","last_became_active":"2024-09-11T05:51:05.499998+0000","last_became_peered":"2024-09-11T05:51:05.499998+0000","last_unstale":"2024-09-11T05:51:14.565539+0000","last_undegraded":"2024-09-11T05:51:14.565539+0000","last_fullsized":"2024-09-11T05:51:14.565539+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:12.062462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":1589248,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":38,"num_write_kb":1532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.603647+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:51:14.603647+0000","last_peered":"2024-09-11T05:51:14.603647+0000","last_clean":"2024-09-11T05:51:14.603647+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:51:14.603647+0000","last_undegraded":"2024-09-11T05:51:14.603647+0000","last_fullsized":"2024-09-11T05:51:14.603647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'5","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:13.602790+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:51:13.602790+0000","last_peered":"2024-09-11T05:51:13.602790+0000","last_clean":"2024-09-11T05:51:13.602790+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:51:13.602790+0000","last_undegraded":"2024-09-11T05:51:13.602790+0000","last_fullsized":"2024-09-11T05:51:13.602790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:29.902378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":39,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575029+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:51:14.575029+0000","last_peered":"2024-09-11T05:51:14.575029+0000","last_clean":"2024-09-11T05:51:14.575029+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:51:14.575029+0000","last_undegraded":"2024-09-11T05:51:14.575029+0000","last_fullsized":"2024-09-11T05:51:14.575029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"31'19","reported_seq":117,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.603801+0000","last_change":"2024-09-11T05:51:07.514746+0000","last_active":"2024-09-11T05:51:14.603801+0000","last_peered":"2024-09-11T05:51:14.603801+0000","last_clean":"2024-09-11T05:51:14.603801+0000","last_became_active":"2024-09-11T05:51:06.470494+0000","last_became_peered":"2024-09-11T05:51:06.470494+0000","last_unstale":"2024-09-11T05:51:14.603801+0000","last_undegraded":"2024-09-11T05:51:14.603801+0000","last_fullsized":"2024-09-11T05:51:14.603801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:28.455054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":622592,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":608,"num_write":18,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.603633+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:51:14.603633+0000","last_peered":"2024-09-11T05:51:14.603633+0000","last_clean":"2024-09-11T05:51:14.603633+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:51:14.603633+0000","last_undegraded":"2024-09-11T05:51:14.603633+0000","last_fullsized":"2024-09-11T05:51:14.603633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"32'80","reported_seq":687,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.832893+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:51:14.832893+0000","last_peered":"2024-09-11T05:51:14.832893+0000","last_clean":"2024-09-11T05:51:14.832893+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:51:14.832893+0000","last_undegraded":"2024-09-11T05:51:14.832893+0000","last_fullsized":"2024-09-11T05:51:14.832893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:19:59.383821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":622608,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1557,"num_read_kb":1912,"num_write":73,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"31'12","reported_seq":110,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.558216+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:51:14.558216+0000","last_peered":"2024-09-11T05:51:14.558216+0000","last_clean":"2024-09-11T05:51:14.558216+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:51:14.558216+0000","last_undegraded":"2024-09-11T05:51:14.558216+0000","last_fullsized":"2024-09-11T05:51:14.558216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:56.687315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":950272,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":928,"num_write":11,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.583994+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:51:14.583994+0000","last_peered":"2024-09-11T05:51:14.583994+0000","last_clean":"2024-09-11T05:51:14.583994+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:51:14.583994+0000","last_undegraded":"2024-09-11T05:51:14.583994+0000","last_fullsized":"2024-09-11T05:51:14.583994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"31'5","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.603847+0000","last_change":"2024-09-11T05:51:07.514375+0000","last_active":"2024-09-11T05:51:14.603847+0000","last_peered":"2024-09-11T05:51:14.603847+0000","last_clean":"2024-09-11T05:51:14.603847+0000","last_became_active":"2024-09-11T05:51:06.471743+0000","last_became_peered":"2024-09-11T05:51:06.471743+0000","last_unstale":"2024-09-11T05:51:14.603847+0000","last_undegraded":"2024-09-11T05:51:14.603847+0000","last_fullsized":"2024-09-11T05:51:14.603847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:38.598026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":5,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.557534+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:51:14.557534+0000","last_peered":"2024-09-11T05:51:14.557534+0000","last_clean":"2024-09-11T05:51:14.557534+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:51:14.557534+0000","last_undegraded":"2024-09-11T05:51:14.557534+0000","last_fullsized":"2024-09-11T05:51:14.557534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"32'6","reported_seq":58,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.737256+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:51:14.737256+0000","last_peered":"2024-09-11T05:51:14.737256+0000","last_clean":"2024-09-11T05:51:14.737256+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:51:14.737256+0000","last_undegraded":"2024-09-11T05:51:14.737256+0000","last_fullsized":"2024-09-11T05:51:14.737256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:31:57.897362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":53,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.574591+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:51:14.574591+0000","last_peered":"2024-09-11T05:51:14.574591+0000","last_clean":"2024-09-11T05:51:14.574591+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:51:14.574591+0000","last_undegraded":"2024-09-11T05:51:14.574591+0000","last_fullsized":"2024-09-11T05:51:14.574591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"31'6","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:13.543542+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:51:13.543542+0000","last_peered":"2024-09-11T05:51:13.543542+0000","last_clean":"2024-09-11T05:51:13.543542+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:51:13.543542+0000","last_undegraded":"2024-09-11T05:51:13.543542+0000","last_fullsized":"2024-09-11T05:51:13.543542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:41:28.670021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":589840,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.584010+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:51:14.584010+0000","last_peered":"2024-09-11T05:51:14.584010+0000","last_clean":"2024-09-11T05:51:14.584010+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:51:14.584010+0000","last_undegraded":"2024-09-11T05:51:14.584010+0000","last_fullsized":"2024-09-11T05:51:14.584010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.574665+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:51:14.574665+0000","last_peered":"2024-09-11T05:51:14.574665+0000","last_clean":"2024-09-11T05:51:14.574665+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:51:14.574665+0000","last_undegraded":"2024-09-11T05:51:14.574665+0000","last_fullsized":"2024-09-11T05:51:14.574665+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"30'10","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:13.543352+0000","last_change":"2024-09-11T05:51:07.507117+0000","last_active":"2024-09-11T05:51:13.543352+0000","last_peered":"2024-09-11T05:51:13.543352+0000","last_clean":"2024-09-11T05:51:13.543352+0000","last_became_active":"2024-09-11T05:51:05.508345+0000","last_became_peered":"2024-09-11T05:51:05.508345+0000","last_unstale":"2024-09-11T05:51:13.543352+0000","last_undegraded":"2024-09-11T05:51:13.543352+0000","last_fullsized":"2024-09-11T05:51:13.543352+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:28:56.584058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"31'4","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.557887+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:51:14.557887+0000","last_peered":"2024-09-11T05:51:14.557887+0000","last_clean":"2024-09-11T05:51:14.557887+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:51:14.557887+0000","last_undegraded":"2024-09-11T05:51:14.557887+0000","last_fullsized":"2024-09-11T05:51:14.557887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:50:44.423384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"32'12","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.680161+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:51:14.680161+0000","last_peered":"2024-09-11T05:51:14.680161+0000","last_clean":"2024-09-11T05:51:14.680161+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:51:14.680161+0000","last_undegraded":"2024-09-11T05:51:14.680161+0000","last_fullsized":"2024-09-11T05:51:14.680161+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:52.906605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1589248,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":22,"num_write_kb":1099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"29'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T05:51:13.543615+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:51:13.543615+0000","last_peered":"2024-09-11T05:51:13.543615+0000","last_clean":"2024-09-11T05:51:13.543615+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:51:13.543615+0000","last_undegraded":"2024-09-11T05:51:13.543615+0000","last_fullsized":"2024-09-11T05:51:13.543615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:15.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":926743,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":55,"num_write":8,"num_write_kb":250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"30'6","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.584262+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:51:14.584262+0000","last_peered":"2024-09-11T05:51:14.584262+0000","last_clean":"2024-09-11T05:51:14.584262+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:51:14.584262+0000","last_undegraded":"2024-09-11T05:51:14.584262+0000","last_fullsized":"2024-09-11T05:51:14.584262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:51.514965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"31'8","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575078+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:51:14.575078+0000","last_peered":"2024-09-11T05:51:14.575078+0000","last_clean":"2024-09-11T05:51:14.575078+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:51:14.575078+0000","last_undegraded":"2024-09-11T05:51:14.575078+0000","last_fullsized":"2024-09-11T05:51:14.575078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:48.622571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"32'22","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575379+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:51:14.575379+0000","last_peered":"2024-09-11T05:51:14.575379+0000","last_clean":"2024-09-11T05:51:14.575379+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:51:14.575379+0000","last_undegraded":"2024-09-11T05:51:14.575379+0000","last_fullsized":"2024-09-11T05:51:14.575379+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:31:59.380672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":1572864,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"32'3","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575660+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:51:14.575660+0000","last_peered":"2024-09-11T05:51:14.575660+0000","last_clean":"2024-09-11T05:51:14.575660+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:51:14.575660+0000","last_undegraded":"2024-09-11T05:51:14.575660+0000","last_fullsized":"2024-09-11T05:51:14.575660+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:27:37.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":37,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"31'2","reported_seq":22,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.565961+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:51:14.565961+0000","last_peered":"2024-09-11T05:51:14.565961+0000","last_clean":"2024-09-11T05:51:14.565961+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:51:14.565961+0000","last_undegraded":"2024-09-11T05:51:14.565961+0000","last_fullsized":"2024-09-11T05:51:14.565961+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:48.189075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"31'18","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.574876+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:51:14.574876+0000","last_peered":"2024-09-11T05:51:14.574876+0000","last_clean":"2024-09-11T05:51:14.574876+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:51:14.574876+0000","last_undegraded":"2024-09-11T05:51:14.574876+0000","last_fullsized":"2024-09-11T05:51:14.574876+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:25:36.896560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"30'191","reported_seq":227,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.565448+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:51:14.565448+0000","last_peered":"2024-09-11T05:51:14.565448+0000","last_clean":"2024-09-11T05:51:14.565448+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:51:14.565448+0000","last_undegraded":"2024-09-11T05:51:14.565448+0000","last_fullsized":"2024-09-11T05:51:14.565448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:02.149730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":47,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":891,"num_read_kb":708,"num_write":306,"num_write_kb":420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"31'19","reported_seq":52,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T05:51:14.575160+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:51:14.575160+0000","last_peered":"2024-09-11T05:51:14.575160+0000","last_clean":"2024-09-11T05:51:14.575160+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:51:14.575160+0000","last_undegraded":"2024-09-11T05:51:14.575160+0000","last_fullsized":"2024-09-11T05:51:14.575160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:56:53.932992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19,"num_write_kb":265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":13363135,"num_objects":44,"num_object_clones":13,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":4437,"num_read_kb":8748,"num_write":944,"num_write_kb":7063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7909376,"data_stored":14757523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":734,"ondisk_log_size":734,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33180,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338660,"statfs":{"total":96636764160,"available":96602787840,"internally_reserved":0,"allocated":2043904,"data_stored":2809871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":147786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":389120,"data_stored":190131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1527808,"data_stored":1298523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":405504,"data_stored":187589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":339968,"data_stored":131418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29484,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342356,"statfs":{"total":96636764160,"available":96606572544,"internally_reserved":0,"allocated":2514944,"data_stored":2287304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2154496,"data_stored":1944276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1228800,"data_stored":2205745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":4118162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":56187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2207744,"data_stored":4172373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":58726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1101824,"data_stored":2081930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:15.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:15.969 INFO:teuthology.orchestra.run.smithi165.stdout:565 write 0x6300ff2 thru 0x630cd0a (0xbd19 bytes) 2024-09-11T05:51:15.977 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:15.985 INFO:teuthology.orchestra.run.smithi165.stdout:567 trunc from 0x630cd0b to 0xb0f1d4 2024-09-11T05:51:16.020 INFO:teuthology.orchestra.run.smithi165.stdout:568 trunc from 0xb0f1d4 to 0xaccd19d 2024-09-11T05:51:16.029 INFO:teuthology.orchestra.run.smithi165.stdout:569 trunc from 0xaccd19d to 0xd47da77 2024-09-11T05:51:16.035 INFO:teuthology.orchestra.run.smithi165.stdout:570 read 0x3f7954f thru 0x3f84b99 (0xb64b bytes) 2024-09-11T05:51:16.035 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.035 INFO:teuthology.orchestra.run.smithi165.stdout:572 read 0x21f7216 thru 0x21fdd02 (0x6aed bytes) 2024-09-11T05:51:16.035 INFO:teuthology.orchestra.run.smithi165.stdout:573 punch from 0x3a6032e to 0x3a6979f, (0x9471 bytes) 2024-09-11T05:51:16.035 INFO:teuthology.orchestra.run.smithi165.stdout:574 write 0xb364898 thru 0xb367231 (0x299a bytes) 2024-09-11T05:51:16.036 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:16.038+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:16.039 INFO:teuthology.orchestra.run.smithi165.stdout:575 read 0x621d832 thru 0x6226d6d (0x953c bytes) 2024-09-11T05:51:16.039 INFO:teuthology.orchestra.run.smithi165.stdout:576 read 0xcb2d48b thru 0xcb3605b (0x8bd1 bytes) 2024-09-11T05:51:16.039 INFO:teuthology.orchestra.run.smithi165.stdout:577 read 0x134d4cb thru 0x1350832 (0x3368 bytes) 2024-09-11T05:51:16.039 INFO:teuthology.orchestra.run.smithi165.stdout:578 read 0x7766d03 thru 0x77744f4 (0xd7f2 bytes) 2024-09-11T05:51:16.045 INFO:teuthology.orchestra.run.smithi165.stdout:579 writesame 0xd99980e thru 0xd99a890 data_size 0x1(0x1083 bytes) 2024-09-11T05:51:16.045 INFO:teuthology.orchestra.run.smithi165.stdout:580 write 0xc97fe99 thru 0xc987255 (0x73bd bytes) 2024-09-11T05:51:16.051 INFO:teuthology.orchestra.run.smithi165.stdout:581 writesame 0xa16a91d thru 0xa178beb data_size 0x1(0xe2cf bytes) 2024-09-11T05:51:16.083 INFO:teuthology.orchestra.run.smithi165.stdout:583 trunc from 0xd99a891 to 0x77f79a9 2024-09-11T05:51:16.111 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:16.112 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:16.113 INFO:teuthology.orchestra.run.smithi165.stdout:584 trunc from 0x77f79a9 to 0x1655333 2024-09-11T05:51:16.123 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:16.123 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:16.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:16.141 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:16.141 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:16.145 INFO:teuthology.orchestra.run.smithi165.stdout:585 write 0x69f90a1 thru 0x6a06309 (0xd269 bytes) 2024-09-11T05:51:16.160 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:16.160 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:16.171 INFO:teuthology.orchestra.run.smithi165.stdout:588 write 0xc307a49 thru 0xc30afee (0x35a6 bytes) 2024-09-11T05:51:16.172 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:16.174+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:16.175 INFO:teuthology.orchestra.run.smithi165.stdout:589 punch from 0x213b4b3 to 0x2145b88, (0xa6d5 bytes) 2024-09-11T05:51:16.175 INFO:teuthology.orchestra.run.smithi165.stdout:590 write 0x1d393da thru 0x1d40d1c (0x7943 bytes) 2024-09-11T05:51:16.178 INFO:teuthology.orchestra.run.smithi165.stdout:591 punch from 0xa2b973e to 0xa2c5523, (0xbde5 bytes) 2024-09-11T05:51:16.178 INFO:teuthology.orchestra.run.smithi165.stdout:592 read 0x8d47860 thru 0x8d4c109 (0x48aa bytes) 2024-09-11T05:51:16.179 INFO:teuthology.orchestra.run.smithi165.stdout:593 trunc from 0xc30afef to 0xb65302b 2024-09-11T05:51:16.180 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:16.180 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:16.199 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:16.199 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:16.200 INFO:teuthology.orchestra.run.smithi165.stdout:594 trunc from 0xb65302b to 0x2643cc6 2024-09-11T05:51:16.221 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:16.221 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:16.231 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:16.230+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:16.232 INFO:teuthology.orchestra.run.smithi165.stdout:595 write 0x5f73ac5 thru 0x5f7c2f8 (0x8834 bytes) 2024-09-11T05:51:16.239 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.245 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:16.245 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:16.249 INFO:teuthology.orchestra.run.smithi165.stdout:597 write 0x7cf7819 thru 0x7d02dec (0xb5d4 bytes) 2024-09-11T05:51:16.259 INFO:teuthology.orchestra.run.smithi165.stdout:598 writesame 0x59a07b1 thru 0x59a2125 data_size 0x13(0x1975 bytes) 2024-09-11T05:51:16.265 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:16.265 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:16.271 INFO:teuthology.orchestra.run.smithi165.stdout:599 trunc from 0x7d02ded to 0xc18656b 2024-09-11T05:51:16.280 INFO:teuthology.orchestra.run.smithi165.stdout:600 write 0x5326ea thru 0x53ff82 (0xd899 bytes) 2024-09-11T05:51:16.282 INFO:teuthology.orchestra.run.smithi165.stdout:601 read 0x6868499 thru 0x68702d3 (0x7e3b bytes) 2024-09-11T05:51:16.283 INFO:teuthology.orchestra.run.smithi165.stdout:602 read 0x3058d00 thru 0x3068692 (0xf993 bytes) 2024-09-11T05:51:16.283 INFO:teuthology.orchestra.run.smithi165.stdout:604 read 0x826e31c thru 0x82703cf (0x20b4 bytes) 2024-09-11T05:51:16.298 INFO:teuthology.orchestra.run.smithi165.stdout:605 trunc from 0xc18656b to 0x23bb074 2024-09-11T05:51:16.305 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:16.305 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:16.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:16.305 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:16.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.306 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.306 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:16.306 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.123476433, 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011567851000000001, 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179399+0000", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179401+0000", 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.573e-06 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179413+0000", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2670000000000001e-05 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179469+0000", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5903000000000003e-05 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179485+0000", 2024-09-11T05:51:16.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.186952+0000", 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074664290000000001 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187123+0000", 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017143900000000001 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187336+0000", 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000212692 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190287+0000", 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029509169999999999 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190763+0000", 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000475799 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190894+0000", 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013149599999999999 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190967+0000", 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2682999999999995e-05 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.123360771000002, 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011710134000000001, 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179515+0000", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179516+0000", 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:51:16.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179529+0000", 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2204e-05 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179555+0000", 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6602999999999999e-05 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.179576+0000", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0636999999999999e-05 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.187937+0000", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083613920000000005 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188065+0000", 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127532 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.188273+0000", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020859200000000001 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.190983+0000", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027099189999999999 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191042+0000", 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9008999999999997e-05 2024-09-11T05:51:16.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191159+0000", 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116556 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.191225+0000", 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6060000000000001e-05 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.027663572000002, 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011718055999999999, 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 87 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275212+0000", 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275213+0000", 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7300000000000005e-07 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275308+0000", 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5353000000000007e-05 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275320+0000", 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1746000000000001e-05 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275328+0000", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2770000000000005e-06 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.282992+0000", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007663998 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283142+0000", 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015011899999999999 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.283361+0000", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000218488 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286081+0000", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027206890000000001 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286782+0000", 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070088400000000003 2024-09-11T05:51:16.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286863+0000", 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1225999999999998e-05 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.286930+0000", 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6502999999999997e-05 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.027528496999999, 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013655390999999999, 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 88 2024-09-11T05:51:16.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275347+0000", 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275348+0000", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4e-07 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275354+0000", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.947e-06 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275367+0000", 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2952999999999999e-05 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.275375+0000", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2230000000000007e-06 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284632+0000", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092572620000000005 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.284856+0000", 2024-09-11T05:51:16.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022421799999999999 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.285198+0000", 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034152800000000001 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.287353+0000", 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021548800000000001 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288869+0000", 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001515728 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.288944+0000", 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5339999999999994e-05 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.289002+0000", 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8471999999999999e-05 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.866277218, 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013083005999999999, 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.332 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436598+0000", 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436600+0000", 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.553e-06 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436737+0000", 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436759+0000", 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1946e-05 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436775+0000", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5767e-05 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444423+0000", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007648874 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444555+0000", 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013180900000000001 2024-09-11T05:51:16.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.444751+0000", 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019567200000000001 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.446939+0000", 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021876830000000002 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449422+0000", 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002483791 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449562+0000", 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013971900000000001 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449681+0000", 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119126 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.86606978, 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014821173, 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436806+0000", 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436807+0000", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.637e-06 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436819+0000", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1663e-05 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436866+0000", 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6879999999999998e-05 2024-09-11T05:51:16.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436885+0000", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9483e-05 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447203+0000", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010317244 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447333+0000", 2024-09-11T05:51:16.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013070200000000001 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.447532+0000", 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019884199999999999 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451135+0000", 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036030820000000001 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451358+0000", 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022260800000000001 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451488+0000", 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130273 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451627+0000", 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013875899999999999 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865943904000002, 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017261187000000001, 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436932+0000", 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436933+0000", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436945+0000", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1939999999999999e-05 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436966+0000", 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0979999999999999e-05 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.436981+0000", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5353e-05 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448139+0000", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011157373999999999 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448357+0000", 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021852500000000001 2024-09-11T05:51:16.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.448717+0000", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451314+0000", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025974359999999998 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453934+0000", 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026193599999999998 2024-09-11T05:51:16.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454076+0000", 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014191599999999999 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454193+0000", 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117199 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865865878000001, 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017801221999999998, 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437010+0000", 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437011+0000", 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3999999999999999e-06 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437151+0000", 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140359 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437177+0000", 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437204+0000", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7645999999999999e-05 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449807+0000", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012602197000000001 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.449944+0000", 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013775500000000001 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.450149+0000", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020427499999999999 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454334+0000", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041850480000000002 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454533+0000", 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000198968 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454697+0000", 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164446 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454811+0000", 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113775 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865639680000001, 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017714963, 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437236+0000", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437237+0000", 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5570000000000001e-06 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437249+0000", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1800000000000001e-05 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437278+0000", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8866000000000001e-05 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437295+0000", 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7447000000000001e-05 2024-09-11T05:51:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451717+0000", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014421237999999999 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.451859+0000", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014248900000000001 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452057+0000", 2024-09-11T05:51:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019812200000000001 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454491+0000", 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024333940000000002 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454830+0000", 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033955500000000001 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454900+0000", 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0049000000000006e-05 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.454951+0000", 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0445999999999998e-05 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865554073999999, 2024-09-11T05:51:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019404264000000001, 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437321+0000", 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437323+0000", 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437333+0000", 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0499999999999999e-05 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437356+0000", 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2293000000000002e-05 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437369+0000", 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.381e-05 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452677+0000", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015307215000000001 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.452924+0000", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024779199999999999 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.453257+0000", 2024-09-11T05:51:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000332944 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456421+0000", 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003164072 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456460+0000", 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8869999999999999e-05 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456617+0000", 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156305 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456726+0000", 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108993 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865435091999998, 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021955514999999998, 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437440+0000", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437441+0000", 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8999999999999995e-07 2024-09-11T05:51:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437529+0000", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8226999999999994e-05 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437556+0000", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6806000000000001e-05 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437570+0000", 2024-09-11T05:51:16.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4003e-05 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455042+0000", 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017471738000000001 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455198+0000", 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.455397+0000", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019827199999999999 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459080+0000", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036837860000000001 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459121+0000", 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0589000000000001e-05 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459260+0000", 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013900199999999999 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459396+0000", 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135933 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865289353000001, 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022184125999999998, 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437586+0000", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437587+0000", 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3699999999999999e-07 2024-09-11T05:51:16.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437593+0000", 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3829999999999997e-06 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437606+0000", 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2649999999999999e-05 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437613+0000", 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2570000000000001e-06 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456824+0000", 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019210188999999999 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.456953+0000", 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012939200000000001 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457153+0000", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000199891 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459412+0000", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022590000000000002 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459524+0000", 2024-09-11T05:51:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111716 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459657+0000", 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013367199999999999 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459770+0000", 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113039 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.865250252999999, 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022423287, 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437625+0000", 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437626+0000", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2399999999999997e-07 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437632+0000", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0599999999999996e-06 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437642+0000", 2024-09-11T05:51:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5699999999999999e-06 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.437650+0000", 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4689999999999997e-06 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.457786+0000", 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020135574999999999 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458013+0000", 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022734200000000001 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.458415+0000", 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040240300000000002 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459798+0000", 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013821759999999999 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459914+0000", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116773 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.459992+0000", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8059000000000006e-05 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.460048+0000", 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6035999999999997e-05 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.387416515999998, 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013428412000000001, 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-09-11T05:51:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915459+0000", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915460+0000", 2024-09-11T05:51:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4759999999999999e-06 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915673+0000", 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021259499999999999 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915717+0000", 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4206999999999999e-05 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915735+0000", 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8073000000000001e-05 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.921996+0000", 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062606429999999998 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922091+0000", 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4765000000000005e-05 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.922203+0000", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112596 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926133+0000", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003930022 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928670+0000", 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025363069999999998 2024-09-11T05:51:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928761+0000", 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1853000000000003e-05 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928887+0000", 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012587500000000001 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.387105249000001, 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013236257, 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915770+0000", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915772+0000", 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6559999999999999e-06 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915784+0000", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2034e-05 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915809+0000", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4839999999999999e-05 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915829+0000", 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9809999999999998e-05 2024-09-11T05:51:16.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923686+0000", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078569520000000004 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.923818+0000", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132182 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924042+0000", 2024-09-11T05:51:16.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022383799999999999 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926144+0000", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002102348 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928947+0000", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028030070000000002 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928981+0000", 2024-09-11T05:51:16.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4014999999999999e-05 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929006+0000", 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5575000000000002e-05 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:16.397 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.387019070000001, 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013436350999999999, 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915856+0000", 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915858+0000", 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5e-06 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915869+0000", 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1039999999999999e-05 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915890+0000", 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1056999999999998e-05 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915907+0000", 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6898999999999999e-05 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.924934+0000", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090266600000000006 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925029+0000", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5313e-05 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.925151+0000", 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122416 2024-09-11T05:51:16.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926154+0000", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010029559999999999 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929245+0000", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003090939 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929272+0000", 2024-09-11T05:51:16.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6843999999999999e-05 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929293+0000", 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0727e-05 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.386897616999999, 2024-09-11T05:51:16.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016090376999999999, 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915978+0000", 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.915979+0000", 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.623e-06 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916191+0000", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000211385 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916218+0000", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7503000000000001e-05 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916242+0000", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3657000000000001e-05 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926301+0000", 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010058958999999999 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926430+0000", 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129472 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.926644+0000", 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000213332 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929041+0000", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023971750000000001 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932021+0000", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029796509999999998 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932048+0000", 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240999999999999e-05 2024-09-11T05:51:16.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932068+0000", 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0378999999999999e-05 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.38658783, 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016382369000000001, 2024-09-11T05:51:16.409 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916288+0000", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916289+0000", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.747e-06 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916301+0000", 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.201e-05 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916325+0000", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3323e-05 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.916364+0000", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9069999999999997e-05 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.927959+0000", 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011595032 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928047+0000", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7719999999999994e-05 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.928196+0000", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149276 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.929059+0000", 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086328399999999997 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932625+0000", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035663819999999999 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932651+0000", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5732e-05 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:52.932670+0000", 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8793000000000001e-05 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.414 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.652688511000001, 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011210540999999999, 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 158 2024-09-11T05:51:16.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650187+0000", 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650188+0000", 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4300000000000001e-06 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650427+0000", 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023869200000000001 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650453+0000", 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6026000000000001e-05 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650475+0000", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.145e-05 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657215+0000", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067406150000000001 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657330+0000", 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114809 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.657573+0000", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243241 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660296+0000", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027227359999999999 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661256+0000", 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00095981899999999999 2024-09-11T05:51:16.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661329+0000", 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2819999999999997e-05 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.661397+0000", 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8903000000000001e-05 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4176.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.652368509999999, 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013818587, 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4176", 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/1205647348", 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 159 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:16.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650507+0000", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650509+0000", 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.773e-06 2024-09-11T05:51:16.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650520+0000", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1202999999999999e-05 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650544+0000", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4023999999999999e-05 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.650576+0000", 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1922999999999999e-05 2024-09-11T05:51:16.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659257+0000", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086811560000000006 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659395+0000", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013841299999999999 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.659593+0000", 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197748 2024-09-11T05:51:16.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.660722+0000", 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011288680000000001 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664169+0000", 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034466929999999998 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:16.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664260+0000", 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0946000000000006e-05 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:50:53.664326+0000", 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5840000000000007e-05 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:16.426 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:16.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:606 read 0x1a871af thru 0x1a9067d (0x94cf bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:608 write 0xd8b1d67 thru 0xd8b79fc (0x5c96 bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:610 write 0x23679ed thru 0x236b32d (0x3941 bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:611 punch from 0x6af1dfc to 0x6af83ed, (0x65f1 bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:612 punch from 0x3826d83 to 0x382957f, (0x27fc bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:613 punch from 0xce12274 to 0xce1dd4a, (0xbad6 bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:614 read 0x84caa5d thru 0x84d0451 (0x59f5 bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:615 writesame 0xee827b thru 0xef15c9 data_size 0x1(0x934f bytes) 2024-09-11T05:51:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:616 trunc from 0xd8b79fd to 0x119743d 2024-09-11T05:51:16.430 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:16.426+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:16.436 INFO:teuthology.orchestra.run.smithi165.stdout:617 trunc from 0x119743d to 0x82e6497 2024-09-11T05:51:16.448 INFO:teuthology.orchestra.run.smithi165.stdout:618 punch from 0x55006f to 0x5597eb, (0x977c bytes) 2024-09-11T05:51:16.448 INFO:teuthology.orchestra.run.smithi165.stdout:622 writesame 0x3cb72ce thru 0x3cb81c8 data_size 0xd(0xefb bytes) 2024-09-11T05:51:16.456 INFO:teuthology.orchestra.run.smithi165.stdout:624 punch from 0x6e57265 to 0x6e60e01, (0x9b9c bytes) 2024-09-11T05:51:16.457 INFO:teuthology.orchestra.run.smithi165.stdout:626 read 0x4327fc9 thru 0x4331c7c (0x9cb4 bytes) 2024-09-11T05:51:16.457 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.457 INFO:teuthology.orchestra.run.smithi165.stdout:628 write 0x2b4818e thru 0x2b4986f (0x16e2 bytes) 2024-09-11T05:51:16.462 INFO:teuthology.orchestra.run.smithi165.stdout:629 read 0x310f9a2 thru 0x311ca0e (0xd06d bytes) 2024-09-11T05:51:16.463 INFO:teuthology.orchestra.run.smithi165.stdout:630 read 0x389519d thru 0x389b26c (0x60d0 bytes) 2024-09-11T05:51:16.463 INFO:teuthology.orchestra.run.smithi165.stdout:632 read 0x42a658c thru 0x42ad942 (0x73b7 bytes) 2024-09-11T05:51:16.463 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.485 INFO:teuthology.orchestra.run.smithi165.stdout:634 write 0xcfe051c thru 0xcfe0b61 (0x646 bytes) 2024-09-11T05:51:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:637 punch from 0x4849de1 to 0x4854eba, (0xb0d9 bytes) 2024-09-11T05:51:16.489 INFO:teuthology.orchestra.run.smithi165.stdout:640 write 0x8bc460f thru 0x8bc6930 (0x2322 bytes) 2024-09-11T05:51:16.494 INFO:teuthology.orchestra.run.smithi165.stdout:641 trunc from 0xcfe0b62 to 0xd303178 2024-09-11T05:51:16.505 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.505 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.516 INFO:teuthology.orchestra.run.smithi165.stdout:648 writesame 0xdbbc12d thru 0xdbc7431 data_size 0x1(0xb305 bytes) 2024-09-11T05:51:16.518 INFO:teuthology.orchestra.run.smithi165.stdout:650 read 0xd75981f thru 0xd767e74 (0xe656 bytes) 2024-09-11T05:51:16.529 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:16.530+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:16.539 INFO:teuthology.orchestra.run.smithi165.stdout:651 trunc from 0xdbc7432 to 0x4bc82d4 2024-09-11T05:51:16.565 INFO:teuthology.orchestra.run.smithi165.stdout:652 read 0x3aa6d6e thru 0x3aae6d7 (0x796a bytes) 2024-09-11T05:51:16.566 INFO:teuthology.orchestra.run.smithi165.stdout:654 write 0x2542a7 thru 0x261ab9 (0xd813 bytes) 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6188 3.1f 3:f95cde7a:::rbd_data.11249b359231.0000000000000000:head [stat out=16b,write 245760~16384 in=16384b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.568524+0000", 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0011306910000000001, 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001166467, 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6188 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568524+0000", 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568524+0000", 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568526+0000", 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8279999999999999e-06 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568551+0000", 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5218000000000001e-05 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568572+0000", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0825000000000001e-05 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568581+0000", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3109999999999995e-06 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568660+0000", 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8435000000000004e-05 2024-09-11T05:51:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568751+0000", 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568906+0000", 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155289 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6191 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:head [stat out=16b,write 230055~15705 in=15705b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.568621+0000", 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0010336830000000001, 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011912979999999999, 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6191 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568621+0000", 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568621+0000", 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568622+0000", 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6700000000000003e-07 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568636+0000", 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3937999999999999e-05 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568646+0000", 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8339999999999994e-06 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568652+0000", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5969999999999997e-06 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568749+0000", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6472999999999996e-05 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.568858+0000", 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109516 2024-09-11T05:51:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569151+0000", 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029228700000000002 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-11T05:51:16.577 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:16.578 INFO:teuthology.orchestra.run.smithi165.stdout:655 read 0xd6c389 thru 0xd7a9fb (0xe673 bytes) 2024-09-11T05:51:16.578 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:51:16.581 INFO:teuthology.orchestra.run.smithi165.stdout:657 trunc from 0x4bc82d4 to 0x91cb279 2024-09-11T05:51:16.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:16.594 INFO:teuthology.orchestra.run.smithi165.stdout:658 read 0x7a9a3a4 thru 0x7aa912a (0xed87 bytes) 2024-09-11T05:51:16.595 INFO:teuthology.orchestra.run.smithi165.stdout:659 writesame 0x411958e thru 0x4126c29 data_size 0xa4(0xd69c bytes) 2024-09-11T05:51:16.623 INFO:teuthology.orchestra.run.smithi165.stdout:660 write 0xd1f78fe thru 0xd207610 (0xfd13 bytes) 2024-09-11T05:51:16.630 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:16.630+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:16.634 INFO:teuthology.orchestra.run.smithi165.stdout:662 write 0x579bd23 thru 0x57a3ac2 (0x7da0 bytes) 2024-09-11T05:51:16.643 INFO:teuthology.orchestra.run.smithi165.stdout:663 trunc from 0xd207611 to 0xd50b94b 2024-09-11T05:51:16.654 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.668 INFO:teuthology.orchestra.run.smithi165.stdout:665 trunc from 0xd50b94b to 0x49c647a 2024-09-11T05:51:16.694 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.695 INFO:teuthology.orchestra.run.smithi165.stdout:667 read 0x44adcf3 thru 0x44bd578 (0xf886 bytes) 2024-09-11T05:51:16.716 INFO:teuthology.orchestra.run.smithi165.stdout:669 write 0x920f76e thru 0x92199c8 (0xa25b bytes) 2024-09-11T05:51:16.730 INFO:teuthology.orchestra.run.smithi165.stdout:671 trunc from 0x92199c9 to 0xab3d976 2024-09-11T05:51:16.730 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:16.726+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:16.737 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:16.737 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:16.737 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:16.738 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:16.738 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:16.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:16.754 INFO:teuthology.orchestra.run.smithi165.stdout:672 write 0xaddf51d thru 0xade687b (0x735f bytes) 2024-09-11T05:51:16.762 INFO:teuthology.orchestra.run.smithi165.stdout:673 write 0x8f57f0 thru 0x8f776d (0x1f7e bytes) 2024-09-11T05:51:16.764 INFO:teuthology.orchestra.run.smithi165.stdout:675 punch from 0x4c33d9e to 0x4c3de03, (0xa065 bytes) 2024-09-11T05:51:16.775 INFO:teuthology.orchestra.run.smithi165.stdout:677 writesame 0xafd6a97 thru 0xafda5e5 data_size 0x15(0x3b4f bytes) 2024-09-11T05:51:16.775 INFO:teuthology.orchestra.run.smithi165.stdout:678 punch from 0x7d87ce7 to 0x7d8b925, (0x3c3e bytes) 2024-09-11T05:51:16.790 INFO:teuthology.orchestra.run.smithi165.stdout:679 write 0xdeef1f8 thru 0xdefb9b0 (0xc7b9 bytes) 2024-09-11T05:51:16.801 INFO:teuthology.orchestra.run.smithi165.stdout:680 write 0xdbd095 thru 0xdc5372 (0x82de bytes) 2024-09-11T05:51:16.806 INFO:teuthology.orchestra.run.smithi165.stdout:682 read 0x6d3deaa thru 0x6d41bb6 (0x3d0d bytes) 2024-09-11T05:51:16.806 INFO:teuthology.orchestra.run.smithi165.stdout:684 read 0x165cbc0 thru 0x165fd71 (0x31b2 bytes) 2024-09-11T05:51:16.807 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.807 INFO:teuthology.orchestra.run.smithi165.stdout:686 read 0x6508bd thru 0x6549a7 (0x40eb bytes) 2024-09-11T05:51:16.811 INFO:teuthology.orchestra.run.smithi165.stdout:687 trunc from 0xdefb9b1 to 0xbe48eca 2024-09-11T05:51:16.831 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:16.830+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:16.837 INFO:teuthology.orchestra.run.smithi165.stdout:688 punch from 0xa9d33a3 to 0xa9dfd24, (0xc981 bytes) 2024-09-11T05:51:16.837 INFO:teuthology.orchestra.run.smithi165.stdout:689 punch from 0x8d413ee to 0x8d44d46, (0x3958 bytes) 2024-09-11T05:51:16.837 INFO:teuthology.orchestra.run.smithi165.stdout:690 read 0x99c90e0 thru 0x99d7572 (0xe493 bytes) 2024-09-11T05:51:16.837 INFO:teuthology.orchestra.run.smithi165.stdout:692 write 0xb9e761f thru 0xb9f344f (0xbe31 bytes) 2024-09-11T05:51:16.843 INFO:teuthology.orchestra.run.smithi165.stdout:693 read 0x885755 thru 0x891ed0 (0xc77c bytes) 2024-09-11T05:51:16.844 INFO:teuthology.orchestra.run.smithi165.stdout:694 read 0x598d05f thru 0x59978b9 (0xa85b bytes) 2024-09-11T05:51:16.844 INFO:teuthology.orchestra.run.smithi165.stdout:697 writesame 0x545e80a thru 0x5461829 data_size 0x10(0x3020 bytes) 2024-09-11T05:51:16.850 INFO:teuthology.orchestra.run.smithi165.stdout:698 read 0x83f490d thru 0x84034f0 (0xebe4 bytes) 2024-09-11T05:51:16.851 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:16.851 INFO:teuthology.orchestra.run.smithi165.stdout:701 write 0x451c584 thru 0x4525ab7 (0x9534 bytes) 2024-09-11T05:51:16.861 INFO:teuthology.orchestra.run.smithi165.stdout:702 trunc from 0xbe48eca to 0xa7e7ca5 2024-09-11T05:51:16.876 INFO:teuthology.orchestra.run.smithi165.stdout:704 writesame 0x3b26ce8 thru 0x3b3403e data_size 0x3b(0xd357 bytes) 2024-09-11T05:51:16.882 INFO:teuthology.orchestra.run.smithi165.stdout:705 read 0xa4ce38b thru 0xa4d84a4 (0xa11a bytes) 2024-09-11T05:51:16.886 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:16.886 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1063 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.920843+0000", 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9648354690000001, 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089251639999999993, 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.887 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1063 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920843+0000", 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920843+0000", 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920845+0000", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.697e-06 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920855+0000", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920877+0000", 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2266e-05 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.920893+0000", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5594e-05 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921712+0000", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000818996 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921760+0000", 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8813000000000002e-05 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922021+0000", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026072500000000001 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926233+0000", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042119080000000003 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929645+0000", 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034116680000000001 2024-09-11T05:51:16.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929712+0000", 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6771000000000002e-05 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929768+0000", 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6789999999999997e-05 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1064 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9642230979999997, 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01221121, 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1064 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.893 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921457+0000", 2024-09-11T05:51:16.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.108e-06 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921463+0000", 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.981e-06 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921476+0000", 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3149999999999999e-05 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921484+0000", 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0739999999999993e-06 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922559+0000", 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010747560000000001 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922816+0000", 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025728099999999999 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.923880+0000", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010636319999999999 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929525+0000", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056459359999999998 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933368+0000", 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038422479999999999 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933523+0000", 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015488899999999999 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933667+0000", 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144155 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.898 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1065 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9641806310000001, 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012476767, 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1065 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921499+0000", 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1099999999999995e-07 2024-09-11T05:51:16.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921506+0000", 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8259999999999993e-06 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921515+0000", 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921522+0000", 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.924898+0000", 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033757380000000001 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925011+0000", 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113663 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925372+0000", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036081200000000001 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929538+0000", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041663910000000002 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933820+0000", 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042815140000000002 2024-09-11T05:51:16.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933911+0000", 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0989000000000006e-05 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933975+0000", 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3806000000000005e-05 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.904 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1066 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9640864580000001, 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012635749, 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1066 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921593+0000", 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5099999999999998e-07 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921599+0000", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4609999999999999e-06 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921609+0000", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0434e-05 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921617+0000", 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2970000000000004e-06 2024-09-11T05:51:16.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926267+0000", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046498060000000003 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926348+0000", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0680999999999995e-05 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926715+0000", 2024-09-11T05:51:16.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000367525 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932135+0000", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054191999999999999 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934076+0000", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019415319999999999 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934172+0000", 2024-09-11T05:51:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5725999999999998e-05 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934228+0000", 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6135999999999999e-05 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1067 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9639592390000002, 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012883237000000001, 2024-09-11T05:51:16.910 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1067 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921720+0000", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4799999999999997e-07 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921725+0000", 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7450000000000003e-06 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921734+0000", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3079999999999997e-06 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921743+0000", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5609999999999999e-06 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927241+0000", 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054980680000000001 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927487+0000", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000245758 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.928564+0000", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010776990000000001 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933718+0000", 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051533330000000004 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934356+0000", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063792199999999997 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934504+0000", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014797200000000001 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934603+0000", 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9023000000000004e-05 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.915 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1068 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9638768410000003, 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015284591, 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1068 2024-09-11T05:51:16.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.917 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0500000000000003e-07 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921807+0000", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9919999999999998e-06 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921816+0000", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0999999999999993e-06 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921824+0000", 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.255e-06 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929887+0000", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080631929999999997 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.930128+0000", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241318 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.931196+0000", 2024-09-11T05:51:16.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001067709 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934254+0000", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030585059999999999 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.936944+0000", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002689126 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937038+0000", 2024-09-11T05:51:16.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4772999999999995e-05 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937086+0000", 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8013999999999998e-05 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1069 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.963812071, 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017393289999999999, 2024-09-11T05:51:16.924 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1069 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921872+0000", 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6480000000000002e-06 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921880+0000", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9850000000000006e-06 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921887+0000", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932217+0000", 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010329680000000001 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932347+0000", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129701 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932757+0000", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000409904 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934275+0000", 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015181579999999999 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939164+0000", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048894680000000001 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939216+0000", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1721999999999997e-05 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939260+0000", 2024-09-11T05:51:16.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3782000000000002e-05 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1126 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.033608+0000", 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8520701329999998, 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097933359999999997, 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1126 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033608+0000", 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033608+0000", 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033609+0000", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4300000000000002e-07 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033614+0000", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5739999999999997e-06 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033622+0000", 2024-09-11T05:51:16.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2220000000000002e-06 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033629+0000", 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6610000000000003e-06 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034254+0000", 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062488300000000001 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034360+0000", 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106298 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034770+0000", 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040989200000000003 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037073+0000", 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023036570000000002 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043335+0000", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062617560000000003 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043373+0000", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7876000000000001e-05 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043402+0000", 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8773999999999999e-05 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1127 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.033699+0000", 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8519799219999999, 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098932120000000002, 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1127 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033699+0000", 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033699+0000", 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033699+0000", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033705+0000", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6779999999999997e-06 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033713+0000", 2024-09-11T05:51:16.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0870000000000007e-06 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033720+0000", 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4559999999999997e-06 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.035099+0000", 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001378937 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.035149+0000", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0621000000000002e-05 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.035344+0000", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019515900000000001 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037082+0000", 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001737174 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043467+0000", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063858170000000002 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043536+0000", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8204e-05 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043592+0000", 2024-09-11T05:51:16.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6285000000000001e-05 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1128 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.033776+0000", 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8519024879999999, 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099267090000000006, 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1128 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033776+0000", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033776+0000", 2024-09-11T05:51:16.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033777+0000", 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5000000000000002e-07 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033781+0000", 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2019999999999997e-06 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033789+0000", 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9570000000000002e-06 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033795+0000", 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3319999999999997e-06 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.035625+0000", 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001829476 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.035729+0000", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010446999999999999 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.036088+0000", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035863699999999997 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038236+0000", 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00214854 2024-09-11T05:51:16.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043651+0000", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054142299999999999 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043681+0000", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0445000000000001e-05 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043703+0000", 2024-09-11T05:51:16.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.177e-05 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1129 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.033847+0000", 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8518315599999999, 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010010563, 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.948 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1129 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033847+0000", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033847+0000", 2024-09-11T05:51:16.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033848+0000", 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.06e-07 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033853+0000", 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9139999999999999e-06 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033860+0000", 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5820000000000004e-06 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033866+0000", 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0399999999999998e-06 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.036416+0000", 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002549585 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.036470+0000", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4404000000000003e-05 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.036656+0000", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018569600000000001 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039006+0000", 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023495629999999998 2024-09-11T05:51:16.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043756+0000", 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047499689999999997 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043807+0000", 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1282999999999997e-05 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043858+0000", 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0821e-05 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1130 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.033946+0000", 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.851732749, 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099997200000000001, 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1130 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033946+0000", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033946+0000", 2024-09-11T05:51:16.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033947+0000", 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8199999999999999e-07 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033951+0000", 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6809999999999997e-06 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033960+0000", 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7399999999999993e-06 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.033966+0000", 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0260000000000004e-06 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037103+0000", 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031367740000000002 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037183+0000", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0391999999999999e-05 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037378+0000", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000194384 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.040255+0000", 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028776969999999998 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043896+0000", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036405729999999998 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043924+0000", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8056999999999999e-05 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043946+0000", 2024-09-11T05:51:16.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1614e-05 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1131 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034052+0000", 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8516263679999998, 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010898178999999999, 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.960 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1131 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034052+0000", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034052+0000", 2024-09-11T05:51:16.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034053+0000", 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6300000000000004e-07 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034057+0000", 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2679999999999996e-06 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034068+0000", 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866000000000001e-05 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034075+0000", 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4110000000000001e-06 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037637+0000", 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035626630000000002 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037720+0000", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2721999999999998e-05 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.037949+0000", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022928099999999999 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043021+0000", 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050714519999999997 2024-09-11T05:51:16.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044845+0000", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018239230000000001 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044904+0000", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9261000000000001e-05 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.044950+0000", 2024-09-11T05:51:16.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6468999999999998e-05 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1132 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8515640950000001, 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011988883000000001, 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1132 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-07 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034119+0000", 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1919999999999998e-06 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034132+0000", 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3036e-05 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034145+0000", 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2849e-05 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038262+0000", 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041172459999999998 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038341+0000", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8678999999999997e-05 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038722+0000", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038121999999999999 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043113+0000", 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004391112 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045962+0000", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028480300000000001 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046045+0000", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3294999999999994e-05 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046103+0000", 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8597999999999997e-05 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1133 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8515186450000001, 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012144546000000001, 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.972 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1133 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034161+0000", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.02e-07 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034166+0000", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3090000000000003e-06 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034174+0000", 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:51:16.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034180+0000", 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0100000000000001e-06 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039023+0000", 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048429850000000002 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039070+0000", 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7558000000000001e-05 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039252+0000", 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000182051 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043138+0000", 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003885234 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046192+0000", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030545910000000002 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046257+0000", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4529000000000005e-05 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046305+0000", 2024-09-11T05:51:16.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7846000000000002e-05 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1348 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649395+0000", 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.2362834820000002, 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010730170000000001, 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1348 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649395+0000", 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649395+0000", 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649396+0000", 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8200000000000001e-07 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649400+0000", 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9470000000000002e-06 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649408+0000", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0290000000000005e-06 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649414+0000", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0069999999999997e-06 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.653909+0000", 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044950459999999999 2024-09-11T05:51:16.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.654012+0000", 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010307899999999999 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.654470+0000", 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045852499999999999 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657778+0000", 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033080800000000001 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659950+0000", 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021714709999999999 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660010+0000", 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0526000000000003e-05 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660125+0000", 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115078 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1349 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649420+0000", 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.2362586769999999, 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011160570999999999, 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.984 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1349 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649420+0000", 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649420+0000", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649420+0000", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649423+0000", 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.033e-06 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649442+0000", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.819e-05 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649447+0000", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6509999999999998e-06 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.655005+0000", 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055581759999999997 2024-09-11T05:51:16.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.655103+0000", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7750000000000004e-05 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.655589+0000", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048616300000000001 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659884+0000", 2024-09-11T05:51:16.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042944999999999997 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660446+0000", 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056193400000000004 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660530+0000", 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4536000000000005e-05 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660581+0000", 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0300000000000003e-05 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1350 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649475+0000", 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.2362034149999999, 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011314027000000001, 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1350 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649475+0000", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649475+0000", 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649476+0000", 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649489+0000", 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3057000000000001e-05 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649499+0000", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.059e-05 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649512+0000", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2563e-05 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.656609+0000", 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070967249999999999 2024-09-11T05:51:16.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.656727+0000", 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117883 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657281+0000", 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055444700000000001 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:16.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659894+0000", 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026133770000000001 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660695+0000", 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080018600000000004 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660747+0000", 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2617999999999997e-05 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660789+0000", 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2077999999999999e-05 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1351 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.2361558559999999, 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012039169000000001, 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:16.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1351 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4200000000000001e-07 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649526+0000", 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2389999999999998e-06 2024-09-11T05:51:16.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649535+0000", 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2279999999999998e-06 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649540+0000", 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3759999999999999e-06 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:16.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657820+0000", 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082799009999999992 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657923+0000", 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103447 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658388+0000", 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046494 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659901+0000", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015124719999999999 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661457+0000", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001555842 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661517+0000", 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9917999999999998e-05 2024-09-11T05:51:17.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661562+0000", 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5364000000000001e-05 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1352 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.236131758, 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014094584, 2024-09-11T05:51:17.002 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1352 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7099999999999997e-07 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649550+0000", 2024-09-11T05:51:17.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6280000000000001e-06 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649556+0000", 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.246e-06 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649561+0000", 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6630000000000001e-06 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658923+0000", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093620470000000001 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659009+0000", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5964000000000006e-05 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659436+0000", 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042755199999999999 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660650+0000", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001213385 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663539+0000", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002889165 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663595+0000", 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6165999999999997e-05 2024-09-11T05:51:17.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663641+0000", 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6397000000000003e-05 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.008 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.009 INFO:teuthology.orchestra.run.smithi165.stdout:706 writesame 0xbac8bd7 thru 0xbacb0c8 data_size 0x1(0x24f2 bytes) 2024-09-11T05:51:17.009 INFO:teuthology.orchestra.run.smithi165.stdout:707 read 0x87562d1 thru 0x875a35c (0x408c bytes) 2024-09-11T05:51:17.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:17.011 INFO:teuthology.orchestra.run.smithi165.stdout:709 write 0xe7a77bf thru 0xe7aa65b (0x2e9d bytes) 2024-09-11T05:51:17.011 INFO:teuthology.orchestra.run.smithi165.stdout:710 read 0x53f5efb thru 0x53f8b5b (0x2c61 bytes) 2024-09-11T05:51:17.011 INFO:teuthology.orchestra.run.smithi165.stdout:711 read 0x329bb8d thru 0x329f994 (0x3e08 bytes) 2024-09-11T05:51:17.011 INFO:teuthology.orchestra.run.smithi165.stdout:712 trunc from 0xe7aa65c to 0x2d09db5 2024-09-11T05:51:17.012 INFO:teuthology.orchestra.run.smithi165.stdout:714 read 0x262129b thru 0x2628e06 (0x7b6c bytes) 2024-09-11T05:51:17.012 INFO:teuthology.orchestra.run.smithi165.stdout:715 write 0x591602f thru 0x591865f (0x2631 bytes) 2024-09-11T05:51:17.012 INFO:teuthology.orchestra.run.smithi165.stdout:716 writesame 0x41b197 thru 0x41d99c data_size 0x2f(0x2806 bytes) 2024-09-11T05:51:17.012 INFO:teuthology.orchestra.run.smithi165.stdout:717 read 0x3bd2b89 thru 0x3be1a31 (0xeea9 bytes) 2024-09-11T05:51:17.012 INFO:teuthology.orchestra.run.smithi165.stdout:718 write 0x1d65d63 thru 0x1d66c29 (0xec7 bytes) 2024-09-11T05:51:17.012 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:17.010+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:17.019 INFO:teuthology.orchestra.run.smithi165.stdout:720 writesame 0x8840c0a thru 0x8847d2c data_size 0x1(0x7123 bytes) 2024-09-11T05:51:17.022 INFO:teuthology.orchestra.run.smithi165.stdout:722 trunc from 0x8847d2d to 0x9b396f7 2024-09-11T05:51:17.035 INFO:teuthology.orchestra.run.smithi165.stdout:723 read 0x9522cf8 thru 0x9528091 (0x539a bytes) 2024-09-11T05:51:17.045 INFO:teuthology.orchestra.run.smithi165.stdout:724 trunc from 0x9b396f7 to 0x4d05953 2024-09-11T05:51:17.067 INFO:teuthology.orchestra.run.smithi165.stdout:725 read 0x26d3678 thru 0x26db5f7 (0x7f80 bytes) 2024-09-11T05:51:17.095 INFO:teuthology.orchestra.run.smithi165.stdout:726 write 0xdd2fd93 thru 0xdd37547 (0x77b5 bytes) 2024-09-11T05:51:17.104 INFO:teuthology.orchestra.run.smithi165.stdout:727 read 0x406fed2 thru 0x4073ea4 (0x3fd3 bytes) 2024-09-11T05:51:17.104 INFO:teuthology.orchestra.run.smithi165.stdout:728 write 0x9ca1ed5 thru 0x9ca33b3 (0x14df bytes) 2024-09-11T05:51:17.109 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:17.106+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:17.110 INFO:teuthology.orchestra.run.smithi165.stdout:729 write 0x512938e thru 0x512f43a (0x60ad bytes) 2024-09-11T05:51:17.132 INFO:teuthology.orchestra.run.smithi165.stdout:730 trunc from 0xdd37548 to 0x6e33311 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6569 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=44b,call rbd.object_map_resize in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.161327+0000", 2024-09-11T05:51:17.161 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0013091999999999999, 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013646419999999999, 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6569 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161327+0000", 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161327+0000", 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161330+0000", 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1810000000000001e-06 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161346+0000", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5988999999999999e-05 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161380+0000", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3643000000000003e-05 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161402+0000", 2024-09-11T05:51:17.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2005e-05 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161507+0000", 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010490299999999999 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161599+0000", 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1984999999999994e-05 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.161814+0000", 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021513200000000001 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-11T05:51:17.166 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.169 INFO:teuthology.orchestra.run.smithi165.stdout:731 write 0x95baf63 thru 0x95c4971 (0x9a0f bytes) 2024-09-11T05:51:17.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:17.179 INFO:teuthology.orchestra.run.smithi165.stdout:732 write 0x4b28be7 thru 0x4b30a84 (0x7e9e bytes) 2024-09-11T05:51:17.188 INFO:teuthology.orchestra.run.smithi165.stdout:733 write 0x66223e6 thru 0x662c96a (0xa585 bytes) 2024-09-11T05:51:17.197 INFO:teuthology.orchestra.run.smithi165.stdout:734 write 0xc0ec42 thru 0xc13b03 (0x4ec2 bytes) 2024-09-11T05:51:17.201 INFO:teuthology.orchestra.run.smithi165.stdout:735 read 0x1126fa3 thru 0x1135bdd (0xec3b bytes) 2024-09-11T05:51:17.211 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:17.210+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:17.212 INFO:teuthology.orchestra.run.smithi165.stdout:736 trunc from 0x95c4972 to 0x28e0599 2024-09-11T05:51:17.243 INFO:teuthology.orchestra.run.smithi165.stdout:737 read 0x19a150c thru 0x19ab66a (0xa15f bytes) 2024-09-11T05:51:17.245 INFO:teuthology.orchestra.run.smithi165.stdout:739 punch from 0x3df2d5 to 0x3e94a6, (0xa1d1 bytes) 2024-09-11T05:51:17.245 INFO:teuthology.orchestra.run.smithi165.stdout:744 read 0x183664d thru 0x183dfb7 (0x796b bytes) 2024-09-11T05:51:17.246 INFO:teuthology.orchestra.run.smithi165.stdout:746 punch from 0x22aae2c to 0x22ad756, (0x292a bytes) 2024-09-11T05:51:17.246 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.246 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.246 INFO:teuthology.orchestra.run.smithi165.stdout:751 write 0x221a689 thru 0x2225732 (0xb0aa bytes) 2024-09-11T05:51:17.249 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.249 INFO:teuthology.orchestra.run.smithi165.stdout:754 read 0x20931af thru 0x209f352 (0xc1a4 bytes) 2024-09-11T05:51:17.250 INFO:teuthology.orchestra.run.smithi165.stdout:755 punch from 0x1340840 to 0x1347cba, (0x747a bytes) 2024-09-11T05:51:17.263 INFO:teuthology.orchestra.run.smithi165.stdout:756 writesame 0x39d415c thru 0x39da034 data_size 0x1(0x5ed9 bytes) 2024-09-11T05:51:17.266 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.296 INFO:teuthology.orchestra.run.smithi165.stdout:758 writesame 0xce1e164 thru 0xce2567c data_size 0x1(0x7519 bytes) 2024-09-11T05:51:17.296 INFO:teuthology.orchestra.run.smithi165.stdout:759 read 0xc7edb60 thru 0xc7edce3 (0x184 bytes) 2024-09-11T05:51:17.296 INFO:teuthology.orchestra.run.smithi165.stdout:760 punch from 0xe62dce to 0xe6ea06, (0xbc38 bytes) 2024-09-11T05:51:17.296 INFO:teuthology.orchestra.run.smithi165.stdout:761 write 0x87cca6 thru 0x88a455 (0xd7b0 bytes) 2024-09-11T05:51:17.300 INFO:teuthology.orchestra.run.smithi165.stdout:765 read 0x8796471 thru 0x87a5c89 (0xf819 bytes) 2024-09-11T05:51:17.300 INFO:teuthology.orchestra.run.smithi165.stdout:767 read 0xb418483 thru 0xb41a975 (0x24f3 bytes) 2024-09-11T05:51:17.300 INFO:teuthology.orchestra.run.smithi165.stdout:768 write 0x3ffdff4 thru 0x400a2b7 (0xc2c4 bytes) 2024-09-11T05:51:17.304 INFO:teuthology.orchestra.run.smithi165.stdout:769 read 0x9eb21b3 thru 0x9eb2257 (0xa5 bytes) 2024-09-11T05:51:17.304 INFO:teuthology.orchestra.run.smithi165.stdout:770 read 0x4c38b38 thru 0x4c40843 (0x7d0c bytes) 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:772 read 0xa99bd71 thru 0xa99c0c8 (0x358 bytes) 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:773 punch from 0x9cbedd1 to 0x9cc5177, (0x63a6 bytes) 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:775 read 0xcca6117 thru 0xccb183f (0xb729 bytes) 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:776 read 0x42d3450 thru 0x42dc69a (0x924b bytes) 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.305 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.306 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.306 INFO:teuthology.orchestra.run.smithi165.stdout:783 read 0xf5e2fb thru 0xf64096 (0x5d9c bytes) 2024-09-11T05:51:17.306 INFO:teuthology.orchestra.run.smithi165.stdout:784 punch from 0x7d9ec04 to 0x7daa985, (0xbd81 bytes) 2024-09-11T05:51:17.307 INFO:teuthology.orchestra.run.smithi165.stdout:786 read 0xa098067 thru 0xa09e1d7 (0x6171 bytes) 2024-09-11T05:51:17.307 INFO:teuthology.orchestra.run.smithi165.stdout:787 writesame 0x6282e19 thru 0x628c66a data_size 0x86(0x9852 bytes) 2024-09-11T05:51:17.311 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:17.310+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:17.314 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:17.314 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:17.314 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:17.314 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:17.314 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.319 INFO:teuthology.orchestra.run.smithi165.stdout:788 read 0x8b77c0a thru 0x8b85635 (0xda2c bytes) 2024-09-11T05:51:17.319 INFO:teuthology.orchestra.run.smithi165.stdout:789 write 0x1951b95 thru 0x195d2ad (0xb719 bytes) 2024-09-11T05:51:17.324 INFO:teuthology.orchestra.run.smithi165.stdout:790 punch from 0x83fa0e5 to 0x83fe37c, (0x4297 bytes) 2024-09-11T05:51:17.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:17.341 INFO:teuthology.orchestra.run.smithi165.stdout:791 trunc from 0xce2567d to 0x3c32dfb 2024-09-11T05:51:17.368 INFO:teuthology.orchestra.run.smithi165.stdout:792 punch from 0x5f4697 to 0x5fa05a, (0x59c3 bytes) 2024-09-11T05:51:17.368 INFO:teuthology.orchestra.run.smithi165.stdout:793 read 0xad06a9 thru 0xadee51 (0xe7a9 bytes) 2024-09-11T05:51:17.389 INFO:teuthology.orchestra.run.smithi165.stdout:794 write 0x7be3d80 thru 0x7bf1055 (0xd2d6 bytes) 2024-09-11T05:51:17.400 INFO:teuthology.orchestra.run.smithi165.stdout:795 write 0x50180f4 thru 0x5022150 (0xa05d bytes) 2024-09-11T05:51:17.410 INFO:teuthology.orchestra.run.smithi165.stdout:796 punch from 0x32bdfb8 to 0x32cb1df, (0xd227 bytes) 2024-09-11T05:51:17.412 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:17.414+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:17.427 INFO:teuthology.orchestra.run.smithi165.stdout:797 write 0xa7ac43e thru 0xa7b69a1 (0xa564 bytes) 2024-09-11T05:51:17.436 INFO:teuthology.orchestra.run.smithi165.stdout:800 write 0x72231e0 thru 0x7223a28 (0x849 bytes) 2024-09-11T05:51:17.442 INFO:teuthology.orchestra.run.smithi165.stdout:801 punch from 0x2907129 to 0x290d37f, (0x6256 bytes) 2024-09-11T05:51:17.442 INFO:teuthology.orchestra.run.smithi165.stdout:803 punch from 0x3ae8ecf to 0x3aeee6d, (0x5f9e bytes) 2024-09-11T05:51:17.442 INFO:teuthology.orchestra.run.smithi165.stdout:804 punch from 0x7780759 to 0x7790437, (0xfcde bytes) 2024-09-11T05:51:17.442 INFO:teuthology.orchestra.run.smithi165.stdout:805 read 0x1a99045 thru 0x1aa6afe (0xdaba bytes) 2024-09-11T05:51:17.443 INFO:teuthology.orchestra.run.smithi165.stdout:806 write 0x2d1350a thru 0x2d16745 (0x323c bytes) 2024-09-11T05:51:17.446 INFO:teuthology.orchestra.run.smithi165.stdout:808 punch from 0x8852d39 to 0x8852f0f, (0x1d6 bytes) 2024-09-11T05:51:17.446 INFO:teuthology.orchestra.run.smithi165.stdout:809 write 0x7358f18 thru 0x7368b7f (0xfc68 bytes) 2024-09-11T05:51:17.457 INFO:teuthology.orchestra.run.smithi165.stdout:811 read 0x291e108 thru 0x292918e (0xb087 bytes) 2024-09-11T05:51:17.458 INFO:teuthology.orchestra.run.smithi165.stdout:813 punch from 0x26625e2 to 0x26636ab, (0x10c9 bytes) 2024-09-11T05:51:17.458 INFO:teuthology.orchestra.run.smithi165.stdout:815 write 0x4cf2d4b thru 0x4cf51a5 (0x245b bytes) 2024-09-11T05:51:17.473 INFO:teuthology.orchestra.run.smithi165.stdout:816 write 0xadd7474 thru 0xaddef27 (0x7ab4 bytes) 2024-09-11T05:51:17.483 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:17.483 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:17.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:17.483 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:953 3.19 3:9b18f687:::rbd_header.112475afb12f:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8711659300000001, 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069395289999999998, 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 953 2024-09-11T05:51:17.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611793+0000", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611810+0000", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6133999999999998e-05 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611844+0000", 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4403000000000002e-05 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611861+0000", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7246999999999999e-05 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611991+0000", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012932899999999999 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.612003+0000", 2024-09-11T05:51:17.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2106000000000001e-05 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616192+0000", 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041892049999999997 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616412+0000", 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021982100000000001 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616807+0000", 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039489700000000001 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618206+0000", 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013989510000000001 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618641+0000", 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043535200000000002 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618695+0000", 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3740999999999999e-05 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618730+0000", 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6023 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.231475385, 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069450629999999996, 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6023 2024-09-11T05:51:17.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251482+0000", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7800000000000001e-07 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251486+0000", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.031e-06 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251496+0000", 2024-09-11T05:51:17.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0326000000000001e-05 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251507+0000", 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-05 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.254914+0000", 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034066790000000001 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.255004+0000", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0777000000000002e-05 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.255303+0000", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029899400000000002 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256292+0000", 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009887889999999999 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258371+0000", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020787620000000001 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258403+0000", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2110999999999998e-05 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258426+0000", 2024-09-11T05:51:17.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2976000000000002e-05 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6094 3.4 3:25de1119:::rbd_data.11249b359231.0000000000000004:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.1204888580000001, 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011224494999999999, 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.497 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6094 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362469+0000", 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362477+0000", 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362496+0000", 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8598999999999999e-05 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362506+0000", 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0336999999999999e-05 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362576+0000", 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0401000000000006e-05 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362671+0000", 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4110999999999994e-05 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364415+0000", 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017441049999999999 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373558+0000", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091431189999999999 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373596+0000", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.803e-05 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373637+0000", 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0706e-05 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373692+0000", 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5531000000000001e-05 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:6094 3.4 e33/23 ondisk, result = 0)", 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.367999+0000", 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.114957083, 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057044660000000001, 2024-09-11T05:51:17.503 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.367999+0000", 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.367999+0000", 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.368000+0000", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8999999999999995e-07 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.368004+0000", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.368018+0000", 2024-09-11T05:51:17.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3339999999999999e-05 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.368027+0000", 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5040000000000008e-06 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373584+0000", 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055567819999999997 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373591+0000", 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.571e-06 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373704+0000", 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112333 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6225 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.507 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.856491055, 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068693469999999996, 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6225 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626466+0000", 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626470+0000", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3950000000000003e-06 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626481+0000", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0178e-05 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626488+0000", 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.807e-06 2024-09-11T05:51:17.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.630692+0000", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042031359999999997 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.630878+0000", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000186775 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.631345+0000", 2024-09-11T05:51:17.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046681899999999999 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.632338+0000", 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099283500000000007 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633182+0000", 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084431000000000003 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633267+0000", 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4973000000000001e-05 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633335+0000", 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7519000000000002e-05 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.513 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6291 3.0 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2719744~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.76060456300000001, 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072388169999999998, 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6291 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722354+0000", 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.81e-06 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722369+0000", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5140000000000001e-05 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722389+0000", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0594e-05 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722399+0000", 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6239999999999997e-06 2024-09-11T05:51:17.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722501+0000", 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101773 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722572+0000", 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1155000000000006e-05 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722694+0000", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121977 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.724008+0000", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013144210000000001 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729476+0000", 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054675929999999998 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729538+0000", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2123000000000001e-05 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729591+0000", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2607e-05 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6289 3.1a e33/23 3:5e4922eb:::rbd_data.11249b359231.000000000000001e:head v 33'20, mlcod=33'18)", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:17.519 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.76013748699999995, 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067092610000000002, 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722820+0000", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2399999999999997e-07 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722835+0000", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4761e-05 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722847+0000", 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2159e-05 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722854+0000", 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722940+0000", 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5834999999999994e-05 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722960+0000", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0171000000000001e-05 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729492+0000", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065318980000000004 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729528+0000", 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6131999999999999e-05 2024-09-11T05:51:17.523 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6339 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.68961279600000003, 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073583650000000004, 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6339 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.58e-07 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793347+0000", 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9529999999999999e-06 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793354+0000", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7870000000000004e-06 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793359+0000", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796483+0000", 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031241120000000001 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796574+0000", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0923999999999999e-05 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796909+0000", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033517300000000002 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.798000+0000", 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001090382 2024-09-11T05:51:17.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800606+0000", 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026059970000000001 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800662+0000", 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6654999999999999e-05 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800702+0000", 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9585000000000001e-05 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6581 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.172445+0000", 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.310511341, 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059031769999999999, 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.530 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6581 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172445+0000", 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172445+0000", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172446+0000", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6799999999999999e-07 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172451+0000", 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5049999999999998e-06 2024-09-11T05:51:17.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172466+0000", 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4163e-05 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.172477+0000", 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1744e-05 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.175106+0000", 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026289180000000001 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.175216+0000", 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109863 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.175641+0000", 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042447399999999999 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.177415+0000", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001773915 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.178201+0000", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000786088 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.178274+0000", 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3855000000000004e-05 2024-09-11T05:51:17.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.178348+0000", 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3783999999999997e-05 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6689 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309246+0000", 2024-09-11T05:51:17.536 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.17371002299999999, 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063165900000000004, 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6689 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309246+0000", 2024-09-11T05:51:17.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309246+0000", 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309248+0000", 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4279999999999999e-06 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309256+0000", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8699999999999992e-06 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309272+0000", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6609999999999999e-05 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309282+0000", 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2639999999999997e-06 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309358+0000", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6335999999999996e-05 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309448+0000", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9925000000000004e-05 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309820+0000", 2024-09-11T05:51:17.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037261100000000001 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315138+0000", 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053173259999999998 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315252+0000", 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011396900000000001 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315430+0000", 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017855200000000001 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315563+0000", 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013269900000000001 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6690 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.542 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.173632916, 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067312420000000001, 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6690 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6199999999999996e-07 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309328+0000", 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0960000000000003e-06 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309338+0000", 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2380000000000003e-06 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309344+0000", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1739999999999997e-06 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310323+0000", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00097914300000000003 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310421+0000", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7985000000000005e-05 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310773+0000", 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000351629 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315744+0000", 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049715580000000001 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315787+0000", 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2413000000000002e-05 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315937+0000", 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000150292 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316055+0000", 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117852 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6691 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.548 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.173571536, 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070656690000000001, 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6691 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309386+0000", 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9800000000000003e-07 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309389+0000", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.614e-06 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309398+0000", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5799999999999992e-06 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309404+0000", 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7180000000000001e-06 2024-09-11T05:51:17.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.311472+0000", 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002068482 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.311638+0000", 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016590999999999999 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.312162+0000", 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052443200000000002 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316079+0000", 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039164250000000003 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316194+0000", 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115456 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316346+0000", 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151955 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316451+0000", 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104299 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6692 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.173529408, 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072663550000000004, 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6692 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.555 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309428+0000", 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309431+0000", 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4079999999999998e-06 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309438+0000", 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.25e-06 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309444+0000", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6319999999999996e-06 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.312810+0000", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003366047 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.313079+0000", 2024-09-11T05:51:17.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026871099999999998 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.314118+0000", 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001039119 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316483+0000", 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002365481 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316579+0000", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5902000000000004e-05 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316650+0000", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1020999999999996e-05 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316693+0000", 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3167999999999997e-05 2024-09-11T05:51:17.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6741 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.091308486999999994, 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074311999999999998, 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6741 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391649+0000", 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8199999999999999e-07 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391653+0000", 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1169999999999997e-06 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391662+0000", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3910000000000004e-06 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391669+0000", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5189999999999998e-06 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.393939+0000", 2024-09-11T05:51:17.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022699920000000002 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394009+0000", 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0530000000000004e-05 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394218+0000", 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020887100000000001 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398527+0000", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043089260000000002 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398947+0000", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042018900000000001 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399034+0000", 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6650999999999994e-05 2024-09-11T05:51:17.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399079+0000", 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5232000000000003e-05 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6742 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.091277721000000006, 2024-09-11T05:51:17.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075653420000000001, 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6742 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5799999999999999e-07 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391683+0000", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.087e-06 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391692+0000", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.507e-06 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391699+0000", 2024-09-11T05:51:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1890000000000001e-06 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394755+0000", 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030557620000000001 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394808+0000", 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2694000000000001e-05 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395007+0000", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001998 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398632+0000", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003624403 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399147+0000", 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051573700000000005 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399203+0000", 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5464999999999998e-05 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399244+0000", 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1140000000000003e-05 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6743 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.091248315999999996, 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077061630000000002, 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6743 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391709+0000", 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7700000000000005e-07 2024-09-11T05:51:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391712+0000", 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391721+0000", 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4530000000000002e-06 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391727+0000", 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4540000000000004e-06 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395418+0000", 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036905359999999999 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395574+0000", 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015623899999999999 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.396102+0000", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000528135 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399304+0000", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032018469999999999 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399323+0000", 2024-09-11T05:51:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9021999999999998e-05 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399366+0000", 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3433999999999998e-05 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399414+0000", 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8056000000000002e-05 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6751 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.402810+0000", 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.080146907000000003, 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057030370000000002, 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6751 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402810+0000", 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402810+0000", 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402810+0000", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7999999999999995e-07 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402814+0000", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.568e-06 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402821+0000", 2024-09-11T05:51:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1579999999999999e-06 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.402827+0000", 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.494e-06 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.404578+0000", 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001750436 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.404666+0000", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8658000000000004e-05 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.405026+0000", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035921 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.405808+0000", 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078264999999999997 2024-09-11T05:51:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.408425+0000", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026170049999999999 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.408479+0000", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3366999999999999e-05 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.408513+0000", 2024-09-11T05:51:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3911000000000001e-05 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6779 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.034257716000000001, 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071484610000000001, 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6779 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448700+0000", 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.33e-07 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448706+0000", 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2339999999999999e-06 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448717+0000", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0767e-05 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448723+0000", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1460000000000001e-06 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.449522+0000", 2024-09-11T05:51:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000798696 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.449659+0000", 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013719900000000001 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.450426+0000", 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076737400000000003 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.452781+0000", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023545739999999999 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455767+0000", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029862320000000001 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455816+0000", 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8646000000000001e-05 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455847+0000", 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1659999999999998e-05 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6781 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.034222619000000003, 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007261254, 2024-09-11T05:51:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6781 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448735+0000", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.16e-07 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448739+0000", 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7840000000000001e-06 2024-09-11T05:51:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448747+0000", 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2609999999999993e-06 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448753+0000", 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0360000000000003e-06 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451009+0000", 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022562770000000001 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451100+0000", 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0957000000000003e-05 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451485+0000", 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038498900000000002 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455598+0000", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041129440000000003 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455899+0000", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030118499999999998 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455956+0000", 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6801e-05 2024-09-11T05:51:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455995+0000", 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9103999999999997e-05 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6802 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.475624+0000", 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0073324180000000003, 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056120099999999997, 2024-09-11T05:51:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6802 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475624+0000", 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475624+0000", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475624+0000", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3000000000000001e-07 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475627+0000", 2024-09-11T05:51:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6869999999999999e-06 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475633+0000", 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8289999999999996e-06 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475637+0000", 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2150000000000003e-06 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.476739+0000", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011016229999999999 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.476841+0000", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102543 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.477114+0000", 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027251100000000002 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.480389+0000", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032755039999999998 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.481132+0000", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074265300000000002 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.481190+0000", 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8112000000000002e-05 2024-09-11T05:51:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.481236+0000", 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5902999999999997e-05 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:17.604 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.605 INFO:teuthology.orchestra.run.smithi165.stdout:819 punch from 0x7b3c216 to 0x7b46123, (0x9f0d bytes) 2024-09-11T05:51:17.605 INFO:teuthology.orchestra.run.smithi165.stdout:820 write 0x72ddcf7 thru 0x72df9dc (0x1ce6 bytes) 2024-09-11T05:51:17.606 INFO:teuthology.orchestra.run.smithi165.stdout:821 write 0x4b6a20a thru 0x4b6b94a (0x1741 bytes) 2024-09-11T05:51:17.606 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:17.608 INFO:teuthology.orchestra.run.smithi165.stdout:829 writesame 0xe213cf4 thru 0xe219503 data_size 0x8(0x5810 bytes) 2024-09-11T05:51:17.608 INFO:teuthology.orchestra.run.smithi165.stdout:830 read 0x182cb73 thru 0x1832ff0 (0x647e bytes) 2024-09-11T05:51:17.608 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.608 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:837 writesame 0x71f5c1d thru 0x71f67be data_size 0x1(0xba2 bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:841 punch from 0xa0fbd9 to 0xa19adc, (0x9f03 bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:843 write 0x746a3b1 thru 0x746bb4c (0x179c bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:844 write 0xa5e59e2 thru 0xa5efcab (0xa2ca bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:845 write 0x3b9e599 thru 0x3ba32f1 (0x4d59 bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:848 writesame 0x1b769cc thru 0x1b7c4ba data_size 0x1(0x5aef bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:850 writesame 0x16f40bc thru 0x16fc9d7 data_size 0x87(0x891c bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:852 punch from 0x2502d50 to 0x250ae37, (0x80e7 bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:853 read 0xa5f174e thru 0xa60097c (0xf22f bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:855 read 0xbed67d8 thru 0xbedb761 (0x4f8a bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:856 read 0x3c522c5 thru 0x3c5cd73 (0xaaaf bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:857 writesame 0x898a299 thru 0x898aab7 data_size 0xb(0x81f bytes) 2024-09-11T05:51:17.609 INFO:teuthology.orchestra.run.smithi165.stdout:859 writesame 0xb1dc74e thru 0xb1e2468 data_size 0x23(0x5d1b bytes) 2024-09-11T05:51:17.610 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:51:17.610 INFO:teuthology.orchestra.run.smithi165.stdout:860 read 0xd248fba thru 0xd24a9d5 (0x1a1c bytes) 2024-09-11T05:51:17.610 INFO:teuthology.orchestra.run.smithi165.stdout:864 punch from 0xd741dfc to 0xd746378, (0x457c bytes) 2024-09-11T05:51:17.611 INFO:teuthology.orchestra.run.smithi165.stdout:865 read 0x903697e thru 0x903c92d (0x5fb0 bytes) 2024-09-11T05:51:17.611 INFO:teuthology.orchestra.run.smithi165.stdout:866 write 0x8b685b5 thru 0x8b73e83 (0xb8cf bytes) 2024-09-11T05:51:17.612 INFO:teuthology.orchestra.run.smithi165.stdout:867 write 0xd1452a3 thru 0xd1452e9 (0x47 bytes) 2024-09-11T05:51:17.612 INFO:teuthology.orchestra.run.smithi165.stdout:868 write 0xbe9711 thru 0xbf38fe (0xa1ee bytes) 2024-09-11T05:51:17.612 INFO:teuthology.orchestra.run.smithi165.stdout:869 write 0x5e2e5d0 thru 0x5e321d5 (0x3c06 bytes) 2024-09-11T05:51:17.612 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:17.606+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:17.622 INFO:teuthology.orchestra.run.smithi165.stdout:871 trunc from 0xe219504 to 0x9797c9b 2024-09-11T05:51:17.649 INFO:teuthology.orchestra.run.smithi165.stdout:872 write 0x8ec7276 thru 0x8ec81e1 (0xf6c bytes) 2024-09-11T05:51:17.654 INFO:teuthology.orchestra.run.smithi165.stdout:874 read 0x24d60f4 thru 0x24d83ab (0x22b8 bytes) 2024-09-11T05:51:17.674 INFO:teuthology.orchestra.run.smithi165.stdout:875 writesame 0xe7dd049 thru 0xe7e5f28 data_size 0x60(0x8ee0 bytes) 2024-09-11T05:51:17.678 INFO:teuthology.orchestra.run.smithi165.stdout:876 trunc from 0xe7e5f29 to 0xcbd7464 2024-09-11T05:51:17.688 INFO:teuthology.orchestra.run.smithi165.stdout:877 punch from 0xc455080 to 0xc45be33, (0x6db3 bytes) 2024-09-11T05:51:17.688 INFO:teuthology.orchestra.run.smithi165.stdout:878 read 0x470a871 thru 0x470c85e (0x1fee bytes) 2024-09-11T05:51:17.697 INFO:teuthology.orchestra.run.smithi165.stdout:879 trunc from 0xcbd7464 to 0x791eadf 2024-09-11T05:51:17.706 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:17.706+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:17.715 INFO:teuthology.orchestra.run.smithi165.stdout:880 read 0x6750917 thru 0x675648c (0x5b76 bytes) 2024-09-11T05:51:17.716 INFO:teuthology.orchestra.run.smithi165.stdout:881 writesame 0x70ee731 thru 0x70f88bc data_size 0xc4(0xa18c bytes) 2024-09-11T05:51:17.721 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.721 INFO:teuthology.orchestra.run.smithi165.stdout:883 writesame 0x1f1dccb thru 0x1f24d8c data_size 0x33(0x70c2 bytes) 2024-09-11T05:51:17.749 INFO:teuthology.orchestra.run.smithi165.stdout:885 writesame 0xd5179c5 thru 0xd51860f data_size 0x1(0xc4b bytes) 2024-09-11T05:51:17.749 INFO:teuthology.orchestra.run.smithi165.stdout:886 read 0xc442f1e thru 0xc44e97b (0xba5e bytes) 2024-09-11T05:51:17.749 INFO:teuthology.orchestra.run.smithi165.stdout:887 writesame 0x861bfad thru 0x862af25 data_size 0x43(0xef79 bytes) 2024-09-11T05:51:17.755 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:17.756 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:17.756 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:17.756 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.758 INFO:teuthology.orchestra.run.smithi165.stdout:888 read 0x82fd5b thru 0x831f08 (0x21ae bytes) 2024-09-11T05:51:17.759 INFO:teuthology.orchestra.run.smithi165.stdout:890 write 0x35d7fa7 thru 0x35d8867 (0x8c1 bytes) 2024-09-11T05:51:17.765 INFO:teuthology.orchestra.run.smithi165.stdout:891 trunc from 0xd518610 to 0xe5ce3c7 2024-09-11T05:51:17.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:17.776 INFO:teuthology.orchestra.run.smithi165.stdout:892 read 0x36a39b3 thru 0x36a6df9 (0x3447 bytes) 2024-09-11T05:51:17.777 INFO:teuthology.orchestra.run.smithi165.stdout:893 writesame 0xa3bd94 thru 0xa48dae data_size 0x5(0xd01b bytes) 2024-09-11T05:51:17.787 INFO:teuthology.orchestra.run.smithi165.stdout:894 write 0xdbcbd21 thru 0xdbd1f92 (0x6272 bytes) 2024-09-11T05:51:17.794 INFO:teuthology.orchestra.run.smithi165.stdout:895 punch from 0x7cbcc38 to 0x7cc4280, (0x7648 bytes) 2024-09-11T05:51:17.794 INFO:teuthology.orchestra.run.smithi165.stdout:897 read 0x40b2e27 thru 0x40b6e9d (0x4077 bytes) 2024-09-11T05:51:17.795 INFO:teuthology.orchestra.run.smithi165.stdout:898 punch from 0x14079ac to 0x140cf2c, (0x5580 bytes) 2024-09-11T05:51:17.796 INFO:teuthology.orchestra.run.smithi165.stdout:899 punch from 0xc35f89f to 0xc360708, (0xe69 bytes) 2024-09-11T05:51:17.796 INFO:teuthology.orchestra.run.smithi165.stdout:900 write 0xa77394c thru 0xa776424 (0x2ad9 bytes) 2024-09-11T05:51:17.802 INFO:teuthology.orchestra.run.smithi165.stdout:901 write 0x5321f41 thru 0x53258d2 (0x3992 bytes) 2024-09-11T05:51:17.807 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:17.806+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:17.812 INFO:teuthology.orchestra.run.smithi165.stdout:903 trunc from 0xe5ce3c7 to 0xa29ddbb 2024-09-11T05:51:17.833 INFO:teuthology.orchestra.run.smithi165.stdout:904 read 0x858b589 thru 0x859021e (0x4c96 bytes) 2024-09-11T05:51:17.834 INFO:teuthology.orchestra.run.smithi165.stdout:905 punch from 0x49ba4f5 to 0x49c2d90, (0x889b bytes) 2024-09-11T05:51:17.840 INFO:teuthology.orchestra.run.smithi165.stdout:906 trunc from 0xa29ddbb to 0x6339fc1 2024-09-11T05:51:17.893 INFO:teuthology.orchestra.run.smithi165.stdout:908 write 0xcc208a2 thru 0xcc26c1f (0x637e bytes) 2024-09-11T05:51:17.901 INFO:teuthology.orchestra.run.smithi165.stdout:910 read 0xb521224 thru 0xb52245b (0x1238 bytes) 2024-09-11T05:51:17.901 INFO:teuthology.orchestra.run.smithi165.stdout:911 read 0xb11050a thru 0xb11c0a4 (0xbb9b bytes) 2024-09-11T05:51:17.901 INFO:teuthology.orchestra.run.smithi165.stdout:912 read 0x28342aa thru 0x2835faf (0x1d06 bytes) 2024-09-11T05:51:17.902 INFO:teuthology.orchestra.run.smithi165.stdout:913 punch from 0x89d61e5 to 0x89d7a89, (0x18a4 bytes) 2024-09-11T05:51:17.902 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:17.907 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:17.907 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:17.907 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:17.907 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:17.907 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:17.907 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:17.906+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:17.916 INFO:teuthology.orchestra.run.smithi165.stdout:916 trunc from 0xcc26c20 to 0x6a14bb3 2024-09-11T05:51:17.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:17.963 INFO:teuthology.orchestra.run.smithi165.stdout:919 writesame 0x9569168 thru 0x9570b7a data_size 0xb(0x7a13 bytes) 2024-09-11T05:51:17.984 INFO:teuthology.orchestra.run.smithi165.stdout:920 trunc from 0x9570b7b to 0x3839ae 2024-09-11T05:51:18.008 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:18.010+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:18.021 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:18.034 INFO:teuthology.orchestra.run.smithi165.stdout:922 writesame 0x90feee thru 0x919a3d data_size 0x1c(0x9b50 bytes) 2024-09-11T05:51:18.037 INFO:teuthology.orchestra.run.smithi165.stdout:923 punch from 0x630236 to 0x6346fd, (0x44c7 bytes) 2024-09-11T05:51:18.037 INFO:teuthology.orchestra.run.smithi165.stdout:924 punch from 0x8be85e to 0x8cb8f7, (0xd099 bytes) 2024-09-11T05:51:18.063 INFO:teuthology.orchestra.run.smithi165.stdout:925 write 0x9277ac2 thru 0x927cdcb (0x530a bytes) 2024-09-11T05:51:18.074 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:18.074 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:18.074 INFO:teuthology.orchestra.run.smithi165.stdout:928 punch from 0x51a9490 to 0x51af7e3, (0x6353 bytes) 2024-09-11T05:51:18.074 INFO:teuthology.orchestra.run.smithi165.stdout:932 write 0x72a2bb1 thru 0x72a3c41 (0x1091 bytes) 2024-09-11T05:51:18.081 INFO:teuthology.orchestra.run.smithi165.stdout:933 read 0x18d7d61 thru 0x18db084 (0x3324 bytes) 2024-09-11T05:51:18.081 INFO:teuthology.orchestra.run.smithi165.stdout:935 read 0x1d928df thru 0x1d9d187 (0xa8a9 bytes) 2024-09-11T05:51:18.081 INFO:teuthology.orchestra.run.smithi165.stdout:937 read 0x189a895 thru 0x18a6f63 (0xc6cf bytes) 2024-09-11T05:51:18.082 INFO:teuthology.orchestra.run.smithi165.stdout:938 read 0x1644393 thru 0x164dd6d (0x99db bytes) 2024-09-11T05:51:18.082 INFO:teuthology.orchestra.run.smithi165.stdout:939 read 0x14e6805 thru 0x14f387f (0xd07b bytes) 2024-09-11T05:51:18.083 INFO:teuthology.orchestra.run.smithi165.stdout:940 read 0x2d065ce thru 0x2d157b0 (0xf1e3 bytes) 2024-09-11T05:51:18.094 INFO:teuthology.orchestra.run.smithi165.stdout:941 write 0x92bb484 thru 0x92c2730 (0x72ad bytes) 2024-09-11T05:51:18.104 INFO:teuthology.orchestra.run.smithi165.stdout:943 write 0x20907b0 thru 0x209c840 (0xc091 bytes) 2024-09-11T05:51:18.109 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:18.110+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:18.109 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:1 2.3 e20/19 2:c4c92e5a:::rbd_trash:head v 20'1, mlcod=0'0)", 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.948116307999999, 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017074670000000001, 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158545+0000", 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4e-07 2024-09-11T05:51:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158565+0000", 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0299999999999999e-05 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158591+0000", 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6146e-05 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158613+0000", 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1647000000000001e-05 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158753+0000", 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014034200000000001 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158836+0000", 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3249000000000002e-05 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160169+0000", 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013321769999999999 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160251+0000", 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2766e-05 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:1212 3.4 e30/23 3:2099f16f:::rbd_data.112475afb12f.00000000000000be:head v 30'17, mlcod=30'15)", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.9106104510000002, 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001644195, 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196050+0000", 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196052+0000", 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7939999999999998e-06 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196072+0000", 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9259e-05 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196121+0000", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-05 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196164+0000", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3016999999999998e-05 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196311+0000", 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147188 2024-09-11T05:51:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.196367+0000", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6597e-05 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.197580+0000", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001212273 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.197694+0000", 2024-09-11T05:51:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114207 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.8947232650000001, 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6026 3.0 e33/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.256467+0000", 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.850192872, 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001621256, 2024-09-11T05:51:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256467+0000", 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256467+0000", 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256469+0000", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.612e-06 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256491+0000", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2381000000000001e-05 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256511+0000", 2024-09-11T05:51:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9545e-05 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256521+0000", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0254999999999999e-05 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256591+0000", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0283000000000005e-05 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256617+0000", 2024-09-11T05:51:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5375999999999999e-05 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258054+0000", 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014369809999999999 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258089+0000", 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4823000000000002e-05 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6053 3.0 e33/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.803258899, 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019874799999999998, 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303403+0000", 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.378e-06 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303414+0000", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.155e-05 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303437+0000", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2342000000000001e-05 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303446+0000", 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6760000000000002e-06 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303513+0000", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6431000000000001e-05 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303536+0000", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3258999999999999e-05 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305349+0000", 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018126640000000001 2024-09-11T05:51:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305389+0000", 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0179999999999998e-05 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6060 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'29, mlcod=33'27)", 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.7924126299999998, 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024674810000000001, 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4099999999999997e-07 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314255+0000", 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314269+0000", 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4177e-05 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314277+0000", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6279999999999996e-06 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314341+0000", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3539000000000003e-05 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314363+0000", 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2272e-05 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316606+0000", 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002243629 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316715+0000", 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108576 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.791733534, 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6091 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.749573568, 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019923990000000002, 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357088+0000", 2024-09-11T05:51:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3829999999999999e-06 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357127+0000", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8819999999999998e-05 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357144+0000", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7027000000000001e-05 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357153+0000", 2024-09-11T05:51:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5810000000000003e-06 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357213+0000", 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9959e-05 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357231+0000", 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7983e-05 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359040+0000", 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018086090000000001 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359079+0000", 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9036999999999999e-05 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.697191498, 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6189 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'20, mlcod=33'18)", 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.569693+0000", 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.536967403, 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015638220000000001, 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569693+0000", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569693+0000", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569694+0000", 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4190000000000001e-06 2024-09-11T05:51:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569731+0000", 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7005000000000002e-05 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569747+0000", 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5386000000000001e-05 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569754+0000", 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4749999999999996e-06 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569854+0000", 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9795000000000003e-05 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.569902+0000", 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7753999999999997e-05 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.571180+0000", 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012782589999999999 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.571257+0000", 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6729000000000003e-05 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.4254442140000001, 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6343 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'23, mlcod=33'21)", 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.3050293390000001, 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001815929, 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801642+0000", 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1019000000000001e-05 2024-09-11T05:51:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801673+0000", 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1349000000000001e-05 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801694+0000", 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1067999999999999e-05 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801706+0000", 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1731e-05 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801771+0000", 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4613e-05 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801789+0000", 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8839e-05 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803393+0000", 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016039299999999999 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803447+0000", 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3380000000000001e-05 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.3014252399999999, 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6354 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'25, mlcod=33'23)", 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.2900520680000001, 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026120029999999999, 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816609+0000", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816617+0000", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7680000000000008e-06 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816630+0000", 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816637+0000", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816690+0000", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2533e-05 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816706+0000", 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6407000000000001e-05 2024-09-11T05:51:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819170+0000", 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024645309999999998 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819220+0000", 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9595999999999997e-05 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.2264330640000001, 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6672 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'39, mlcod=33'37)", 2024-09-11T05:51:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.80787946600000005, 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022554760000000002, 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298783+0000", 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.362e-06 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298845+0000", 2024-09-11T05:51:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1544999999999994e-05 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298878+0000", 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3642000000000001e-05 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298895+0000", 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6750000000000001e-05 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299066+0000", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017087600000000001 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299124+0000", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8213e-05 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.300980+0000", 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00185586 2024-09-11T05:51:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.301036+0000", 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6227999999999998e-05 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.75245794399999999, 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6836 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'30, mlcod=33'28)", 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.562105+0000", 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.54455537499999995, 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001654797, 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562105+0000", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562105+0000", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562107+0000", 2024-09-11T05:51:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.644e-06 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562156+0000", 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8875999999999998e-05 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562193+0000", 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.667e-05 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562209+0000", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5688999999999998e-05 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562353+0000", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014475600000000001 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.562409+0000", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5242999999999999e-05 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.563686+0000", 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012777019999999999 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.563760+0000", 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3217000000000001e-05 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6924 3.0 e34/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 34'25, mlcod=34'23)", 2024-09-11T05:51:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.40522729699999999, 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019233869999999999, 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2e-07 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701440+0000", 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.314e-06 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701448+0000", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1939999999999998e-06 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701474+0000", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6336999999999999e-05 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701570+0000", 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5503999999999999e-05 2024-09-11T05:51:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701588+0000", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8510000000000001e-05 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703303+0000", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017153380000000001 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703356+0000", 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2769999999999998e-05 2024-09-11T05:51:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.25982867999999998, 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:18.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:18.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:18.200 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:18.201 INFO:teuthology.orchestra.run.smithi165.stdout:946 read 0x80822bc thru 0x808f1cc (0xcf11 bytes) 2024-09-11T05:51:18.201 INFO:teuthology.orchestra.run.smithi165.stdout:947 punch from 0x4c1d0af to 0x4c2b06e, (0xdfbf bytes) 2024-09-11T05:51:18.201 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:18.201 INFO:teuthology.orchestra.run.smithi165.stdout:954 read 0x3cd4085 thru 0x3cdb613 (0x758f bytes) 2024-09-11T05:51:18.201 INFO:teuthology.orchestra.run.smithi165.stdout:955 read 0x6c11eda thru 0x6c1b6c5 (0x97ec bytes) 2024-09-11T05:51:18.202 INFO:teuthology.orchestra.run.smithi165.stdout:956 read 0x920607c thru 0x92082d7 (0x225c bytes) 2024-09-11T05:51:18.202 INFO:teuthology.orchestra.run.smithi165.stdout:957 clone 5 order 22 su 65536 sc 10 2024-09-11T05:51:18.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:18.209 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:18.210+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:18.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:18.339 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:18.338+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:18.377 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:18.377 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:18.377 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:18.377 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:18.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:18.435 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:18.434+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:18.442 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:18.442 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:18.459 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:18.459 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:18.476 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:18.476 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:18.493 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:18.493 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:18.511 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:18.510+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:18.514 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:18.514 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:18.535 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:18.535 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:18.552 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:18.553 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:18.555 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:18.556 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:18.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:18.578 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:18.578 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:18.647 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:18.646+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:18.704 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:18.704 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:18.713 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:18.714+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:18.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:18.745 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone4 from 0x92c2731 (overlap 0x3839ae) to 0x2cfaf1 2024-09-11T05:51:18.813 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:18.810+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:18.858 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:18.858 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:18.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:18.920 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:18.922+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:18.934 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-09-11T05:51:18.948 INFO:teuthology.orchestra.run.smithi165.stdout:960 read 0x3c09517 thru 0x3c09b16 (0x600 bytes) 2024-09-11T05:51:18.973 INFO:teuthology.orchestra.run.smithi165.stdout:961 write 0xae4437a thru 0xae44f15 (0xb9c bytes) 2024-09-11T05:51:18.995 INFO:teuthology.orchestra.run.smithi165.stdout:963 write 0xbd9827c thru 0xbda5702 (0xd487 bytes) 2024-09-11T05:51:18.996 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:18.996 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.005 INFO:teuthology.orchestra.run.smithi165.stdout:964 writesame 0x87cf047 thru 0x87db343 data_size 0x7(0xc2fd bytes) 2024-09-11T05:51:19.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:19.016 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:19.014+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:19.069 INFO:teuthology.orchestra.run.smithi165.stdout:966 trunc from 0xbda5703 to 0x34de6c8 2024-09-11T05:51:19.116 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:19.114+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:19.125 INFO:teuthology.orchestra.run.smithi165.stdout:967 write 0x38545f1 thru 0x385ee35 (0xa845 bytes) 2024-09-11T05:51:19.144 INFO:teuthology.orchestra.run.smithi165.stdout:969 trunc from 0x385ee36 to 0x9d5c218 2024-09-11T05:51:19.157 INFO:teuthology.orchestra.run.smithi165.stdout:970 writesame 0x61fda73 thru 0x620b199 data_size 0x1(0xd727 bytes) 2024-09-11T05:51:19.185 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.185 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:19.219 INFO:teuthology.orchestra.run.smithi165.stdout:971 read 0x2a89d48 thru 0x2a921e2 (0x849b bytes) 2024-09-11T05:51:19.221 INFO:teuthology.orchestra.run.smithi165.stdout:972 writesame 0x45b2b22 thru 0x45bd707 data_size 0x1(0xabe6 bytes) 2024-09-11T05:51:19.244 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:19.246+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:19.281 INFO:teuthology.orchestra.run.smithi165.stdout:973 read 0x795b81f thru 0x7965bf4 (0xa3d6 bytes) 2024-09-11T05:51:19.299 INFO:teuthology.orchestra.run.smithi165.stdout:974 write 0xba809c1 thru 0xba8f5dd (0xec1d bytes) 2024-09-11T05:51:19.306 INFO:teuthology.orchestra.run.smithi165.stdout:975 write 0x7a43c29 thru 0x7a4dcf8 (0xa0d0 bytes) 2024-09-11T05:51:19.312 INFO:teuthology.orchestra.run.smithi165.stdout:976 read 0x947df9a thru 0x947ee83 (0xeea bytes) 2024-09-11T05:51:19.313 INFO:teuthology.orchestra.run.smithi165.stdout:977 punch from 0x9226461 to 0x923565e, (0xf1fd bytes) 2024-09-11T05:51:19.313 INFO:teuthology.orchestra.run.smithi165.stdout:978 read 0x7418cb6 thru 0x741b477 (0x27c2 bytes) 2024-09-11T05:51:19.318 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:19.318+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:19.323 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.324 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.328 INFO:teuthology.orchestra.run.smithi165.stdout:979 write 0xc4ef3d1 thru 0xc4f2201 (0x2e31 bytes) 2024-09-11T05:51:19.338 INFO:teuthology.orchestra.run.smithi165.stdout:980 write 0x581c31 thru 0x58bb32 (0x9f02 bytes) 2024-09-11T05:51:19.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:19.401 INFO:teuthology.orchestra.run.smithi165.stdout:981 read 0x8ef4015 thru 0x8f03adb (0xfac7 bytes) 2024-09-11T05:51:19.401 INFO:teuthology.orchestra.run.smithi165.stdout:982 read 0x27cd72 thru 0x28bde6 (0xf075 bytes) 2024-09-11T05:51:19.402 INFO:teuthology.orchestra.run.smithi165.stdout:986 writesame 0x460136e thru 0x4609243 data_size 0x22(0x7ed6 bytes) 2024-09-11T05:51:19.406 INFO:teuthology.orchestra.run.smithi165.stdout:987 writesame 0x545e3d0 thru 0x54622f4 data_size 0x35(0x3f25 bytes) 2024-09-11T05:51:19.416 INFO:teuthology.orchestra.run.smithi165.stdout:989 write 0x6f66856 thru 0x6f69c6c (0x3417 bytes) 2024-09-11T05:51:19.419 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:19.418+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:19.421 INFO:teuthology.orchestra.run.smithi165.stdout:990 write 0x4db0bc2 thru 0x4db5565 (0x49a4 bytes) 2024-09-11T05:51:19.423 INFO:teuthology.orchestra.run.smithi165.stdout:991 read 0xc298daa thru 0xc299854 (0xaab bytes) 2024-09-11T05:51:19.423 INFO:teuthology.orchestra.run.smithi165.stdout:992 write 0x912969b thru 0x9135032 (0xb998 bytes) 2024-09-11T05:51:19.429 INFO:teuthology.orchestra.run.smithi165.stdout:993 read 0x24900fa thru 0x249554c (0x5453 bytes) 2024-09-11T05:51:19.430 INFO:teuthology.orchestra.run.smithi165.stdout:994 read 0xa051bb3 thru 0xa054c51 (0x309f bytes) 2024-09-11T05:51:19.431 INFO:teuthology.orchestra.run.smithi165.stdout:995 writesame 0x13213d thru 0x13e5b1 data_size 0x1(0xc475 bytes) 2024-09-11T05:51:19.520 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:19.522+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:19.522 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.522 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.534 INFO:teuthology.orchestra.run.smithi165.stdout:998 writesame 0x522500f thru 0x523199a data_size 0x2(0xc98c bytes) 2024-09-11T05:51:19.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:19.567 INFO:teuthology.orchestra.run.smithi165.stdout:999 write 0x637c4ef thru 0x6381d65 (0x5877 bytes) 2024-09-11T05:51:19.575 INFO:teuthology.orchestra.run.smithi165.stdout:1000 write 0xc37b950 thru 0xc38345f (0x7b10 bytes) 2024-09-11T05:51:19.586 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.586 INFO:teuthology.orchestra.run.smithi165.stdout:1003 read 0x354bacb thru 0x355b633 (0xfb69 bytes) 2024-09-11T05:51:19.587 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.587 INFO:teuthology.orchestra.run.smithi165.stdout:1006 read 0x2bcf82c thru 0x2bd7138 (0x790d bytes) 2024-09-11T05:51:19.611 INFO:teuthology.orchestra.run.smithi165.stdout:1007 trunc from 0xc4f2202 to 0x36a7ee4 2024-09-11T05:51:19.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:51:19.621 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:19.622+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:19.653 INFO:teuthology.orchestra.run.smithi165.stdout:1008 write 0x3fa4a2f thru 0x3fa4f88 (0x55a bytes) 2024-09-11T05:51:19.662 INFO:teuthology.orchestra.run.smithi165.stdout:1009 trunc from 0x3fa4f89 to 0x6c0d74e 2024-09-11T05:51:19.665 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.665 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.674 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.674 INFO:teuthology.orchestra.run.smithi165.stdout:1011 write 0x1dc16ae thru 0x1dc6d38 (0x568b bytes) 2024-09-11T05:51:19.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:19.706 INFO:teuthology.orchestra.run.smithi165.stdout:1012 writesame 0x4c2f88d thru 0x4c3e2e1 data_size 0x1(0xea55 bytes) 2024-09-11T05:51:19.722 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:19.722+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:19.749 INFO:teuthology.orchestra.run.smithi165.stdout:1013 punch from 0x6541f8b to 0x654ed03, (0xcd78 bytes) 2024-09-11T05:51:19.751 INFO:teuthology.orchestra.run.smithi165.stdout:1014 trunc from 0x6c0d74e to 0x7be45fa 2024-09-11T05:51:19.758 INFO:teuthology.orchestra.run.smithi165.stdout:1015 read 0x5ed30f6 thru 0x5edfa4b (0xc956 bytes) 2024-09-11T05:51:19.758 INFO:teuthology.orchestra.run.smithi165.stdout:1016 write 0x2dbddc1 thru 0x2dc95e9 (0xb829 bytes) 2024-09-11T05:51:19.761 INFO:teuthology.orchestra.run.smithi165.stdout:1017 trunc from 0x7be45fa to 0x886b54e 2024-09-11T05:51:19.767 INFO:teuthology.orchestra.run.smithi165.stdout:1018 read 0x812f752 thru 0x813cbee (0xd49d bytes) 2024-09-11T05:51:19.768 INFO:teuthology.orchestra.run.smithi165.stdout:1020 punch from 0x8a6e04 to 0x8b1bf1, (0xaded bytes) 2024-09-11T05:51:19.768 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.768 INFO:teuthology.orchestra.run.smithi165.stdout:1022 write 0x811edc4 thru 0x8123d02 (0x4f3f bytes) 2024-09-11T05:51:19.776 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.776 INFO:teuthology.orchestra.run.smithi165.stdout:1025 write 0x72a4ee4 thru 0x72ad26c (0x8389 bytes) 2024-09-11T05:51:19.786 INFO:teuthology.orchestra.run.smithi165.stdout:1027 write 0x9f01270 thru 0x9f0a9a7 (0x9738 bytes) 2024-09-11T05:51:19.791 INFO:teuthology.orchestra.run.smithi165.stdout:1028 write 0x5291a3c thru 0x5299047 (0x760c bytes) 2024-09-11T05:51:19.795 INFO:teuthology.orchestra.run.smithi165.stdout:1029 writesame 0x97f1cdb thru 0x97f2637 data_size 0x11(0x95d bytes) 2024-09-11T05:51:19.800 INFO:teuthology.orchestra.run.smithi165.stdout:1030 writesame 0x6f8c4c5 thru 0x6f9856e data_size 0x5b(0xc0aa bytes) 2024-09-11T05:51:19.804 INFO:teuthology.orchestra.run.smithi165.stdout:1031 write 0x3e70c5c thru 0x3e77b55 (0x6efa bytes) 2024-09-11T05:51:19.806 INFO:teuthology.orchestra.run.smithi165.stdout:1032 read 0x10751b7 thru 0x107b8ea (0x6734 bytes) 2024-09-11T05:51:19.814 INFO:teuthology.orchestra.run.smithi165.stdout:1033 write 0xa5de8cf thru 0xa5eaf01 (0xc633 bytes) 2024-09-11T05:51:19.818 INFO:teuthology.orchestra.run.smithi165.stdout:1034 writesame 0x3af090 thru 0x3b8e35 data_size 0x11(0x9da6 bytes) 2024-09-11T05:51:19.823 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:19.822+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:19.824 INFO:teuthology.orchestra.run.smithi165.stdout:1035 punch from 0x87cdb49 to 0x87d93ab, (0xb862 bytes) 2024-09-11T05:51:19.824 INFO:teuthology.orchestra.run.smithi165.stdout:1036 read 0x4b3ec19 thru 0x4b4052e (0x1916 bytes) 2024-09-11T05:51:19.824 INFO:teuthology.orchestra.run.smithi165.stdout:1037 writesame 0x2ae13b8 thru 0x2aef039 data_size 0xa(0xdc82 bytes) 2024-09-11T05:51:19.828 INFO:teuthology.orchestra.run.smithi165.stdout:1038 read 0xd72104 thru 0xd7c504 (0xa401 bytes) 2024-09-11T05:51:19.829 INFO:teuthology.orchestra.run.smithi165.stdout:1039 write 0x1329b3b thru 0x1333a27 (0x9eed bytes) 2024-09-11T05:51:19.855 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.855 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.861 INFO:teuthology.orchestra.run.smithi165.stdout:1040 write 0xca85f15 thru 0xca8d27b (0x7367 bytes) 2024-09-11T05:51:19.867 INFO:teuthology.orchestra.run.smithi165.stdout:1041 write 0x493ff23 thru 0x494079f (0x87d bytes) 2024-09-11T05:51:19.869 INFO:teuthology.orchestra.run.smithi165.stdout:1042 punch from 0x26eec08 to 0x26efa1b, (0xe13 bytes) 2024-09-11T05:51:19.869 INFO:teuthology.orchestra.run.smithi165.stdout:1043 read 0x60c867b thru 0x60d5b4a (0xd4d0 bytes) 2024-09-11T05:51:19.870 INFO:teuthology.orchestra.run.smithi165.stdout:1044 read 0x59cde98 thru 0x59dac8d (0xcdf6 bytes) 2024-09-11T05:51:19.870 INFO:teuthology.orchestra.run.smithi165.stdout:1045 write 0xa680ad thru 0xa75626 (0xd57a bytes) 2024-09-11T05:51:19.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:19.880 INFO:teuthology.orchestra.run.smithi165.stdout:1046 trunc from 0xca8d27c to 0x742018b 2024-09-11T05:51:19.924 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:19.922+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:19.926 INFO:teuthology.orchestra.run.smithi165.stdout:1048 writesame 0xb106fb3 thru 0xb10a9e8 data_size 0x1(0x3a36 bytes) 2024-09-11T05:51:19.930 INFO:teuthology.orchestra.run.smithi165.stdout:1050 trunc from 0xb10a9e9 to 0xd8b4300 2024-09-11T05:51:19.939 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.940 INFO:teuthology.orchestra.run.smithi165.stdout:1052 punch from 0x4af1fc5 to 0x4afcca2, (0xacdd bytes) 2024-09-11T05:51:19.940 INFO:teuthology.orchestra.run.smithi165.stdout:1053 read 0xbbb3d6f thru 0xbbc35eb (0xf87d bytes) 2024-09-11T05:51:19.961 INFO:teuthology.orchestra.run.smithi165.stdout:1055 trunc from 0xd8b4300 to 0x294fb4d 2024-09-11T05:51:19.996 INFO:teuthology.orchestra.run.smithi165.stdout:1056 read 0x28af64d thru 0x28bd961 (0xe315 bytes) 2024-09-11T05:51:19.996 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:19.996 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:19.997 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:19.997 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:20.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:20.017 INFO:teuthology.orchestra.run.smithi165.stdout:1059 write 0x5cfbf15 thru 0x5d03b50 (0x7c3c bytes) 2024-09-11T05:51:20.023 INFO:teuthology.orchestra.run.smithi165.stdout:1060 read 0x2e20a35 thru 0x2e26ce3 (0x62af bytes) 2024-09-11T05:51:20.056 INFO:teuthology.orchestra.run.smithi165.stdout:1063 write 0xe86a9dd thru 0xe874317 (0x993b bytes) 2024-09-11T05:51:20.056 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:20.058+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:20.064 INFO:teuthology.orchestra.run.smithi165.stdout:1064 writesame 0xb57ddc0 thru 0xb58071b data_size 0x2(0x295c bytes) 2024-09-11T05:51:20.071 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:20.071 INFO:teuthology.orchestra.run.smithi165.stdout:1070 write 0xd7d3db1 thru 0xd7e2501 (0xe751 bytes) 2024-09-11T05:51:20.077 INFO:teuthology.orchestra.run.smithi165.stdout:1071 read 0x25458bb thru 0x2550463 (0xaba9 bytes) 2024-09-11T05:51:20.078 INFO:teuthology.orchestra.run.smithi165.stdout:1073 punch from 0x150c988 to 0x15165cd, (0x9c45 bytes) 2024-09-11T05:51:20.078 INFO:teuthology.orchestra.run.smithi165.stdout:1074 read 0x5735e77 thru 0x57458d8 (0xfa62 bytes) 2024-09-11T05:51:20.089 INFO:teuthology.orchestra.run.smithi165.stdout:1075 writesame 0xea2789c thru 0xea35987 data_size 0xa(0xe0ec bytes) 2024-09-11T05:51:20.089 INFO:teuthology.orchestra.run.smithi165.stdout:1076 clone 6 order 22 su 2097152 sc 8 2024-09-11T05:51:20.126 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:20.122+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:20.134 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:20.134 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:20.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:20.240 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:20.242+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:20.318 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:20.318 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:20.327 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:20.326+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:20.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:20.428 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:20.430+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:20.499 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:20.500 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:20.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:20.529 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:20.526+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:20.591 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:51:20.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:20.629 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:20.630+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:20.684 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:20.684 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:20.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:20.738 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone5 from 0xea35988 (overlap 0x294fb4d) to 0x252e22b 2024-09-11T05:51:20.739 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:20.738+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:20.768 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:20.769 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:20.803 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:20.803 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:20.828 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:20.828 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:20.830 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:20.830+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:20.848 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:20.849 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:20.858 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:20.858 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:20.858 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:20.858 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:20.868 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:20.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:20.889 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:20.889 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:20.910 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:20.910 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:20.931 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:20.931 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:20.931 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:20.930+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:20.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:20.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T05:51:21.002 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-09-11T05:51:21.009 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:21.009 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:21.009 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:21.009 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:21.009 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:21.013 INFO:teuthology.orchestra.run.smithi165.stdout:1077 writesame 0x381ceb2 thru 0x382b72f data_size 0x1(0xe87e bytes) 2024-09-11T05:51:21.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:21.032 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:21.034+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:21.058 INFO:teuthology.orchestra.run.smithi165.stdout:1078 punch from 0x638275f to 0x6383573, (0xe14 bytes) 2024-09-11T05:51:21.058 INFO:teuthology.orchestra.run.smithi165.stdout:1081 punch from 0xb26858 to 0xb3371d, (0xcec5 bytes) 2024-09-11T05:51:21.076 INFO:teuthology.orchestra.run.smithi165.stdout:1082 trunc from 0xea35988 to 0x1b3d529 2024-09-11T05:51:21.133 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:21.134+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:21.146 INFO:teuthology.orchestra.run.smithi165.stdout:1083 writesame 0x1e6768e thru 0x1e6c73d data_size 0x8(0x50b0 bytes) 2024-09-11T05:51:21.175 INFO:teuthology.orchestra.run.smithi165.stdout:1085 write 0xbaa195f thru 0xbaa7109 (0x57ab bytes) 2024-09-11T05:51:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:1086 read 0xa2c0da3 thru 0xa2ca983 (0x9be1 bytes) 2024-09-11T05:51:21.182 INFO:teuthology.orchestra.run.smithi165.stdout:1087 read 0x99ec688 thru 0x99f0d1f (0x4698 bytes) 2024-09-11T05:51:21.182 INFO:teuthology.orchestra.run.smithi165.stdout:1088 write 0x2253483 thru 0x225d6a2 (0xa220 bytes) 2024-09-11T05:51:21.188 INFO:teuthology.orchestra.run.smithi165.stdout:1089 write 0xb5f4e96 thru 0xb5fb4b4 (0x661f bytes) 2024-09-11T05:51:21.193 INFO:teuthology.orchestra.run.smithi165.stdout:1093 write 0x8b5dd2a thru 0x8b63900 (0x5bd7 bytes) 2024-09-11T05:51:21.199 INFO:teuthology.orchestra.run.smithi165.stdout:1094 read 0x4f422cb thru 0x4f43d71 (0x1aa7 bytes) 2024-09-11T05:51:21.199 INFO:teuthology.orchestra.run.smithi165.stdout:1095 writesame 0x219084 thru 0x223e68 data_size 0x47(0xade5 bytes) 2024-09-11T05:51:21.219 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:21.219 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:21.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:21.219 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1064 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.2971508810000003, 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01221121, 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1064 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.220 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921457+0000", 2024-09-11T05:51:21.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.108e-06 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921463+0000", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.981e-06 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921476+0000", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3149999999999999e-05 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921484+0000", 2024-09-11T05:51:21.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0739999999999993e-06 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922559+0000", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010747560000000001 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922816+0000", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025728099999999999 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.923880+0000", 2024-09-11T05:51:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010636319999999999 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929525+0000", 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056459359999999998 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933368+0000", 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038422479999999999 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933523+0000", 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015488899999999999 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933667+0000", 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144155 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1065 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.2971084140000002, 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012476767, 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1065 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921499+0000", 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1099999999999995e-07 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921506+0000", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8259999999999993e-06 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921515+0000", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921522+0000", 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.924898+0000", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033757380000000001 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925011+0000", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113663 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925372+0000", 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036081200000000001 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929538+0000", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041663910000000002 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933820+0000", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042815140000000002 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933911+0000", 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0989000000000006e-05 2024-09-11T05:51:21.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933975+0000", 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3806000000000005e-05 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1066 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.2970142409999994, 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012635749, 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.231 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1066 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921593+0000", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5099999999999998e-07 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921599+0000", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4609999999999999e-06 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921609+0000", 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0434e-05 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921617+0000", 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2970000000000004e-06 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926267+0000", 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046498060000000003 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926348+0000", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0680999999999995e-05 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926715+0000", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000367525 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932135+0000", 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054191999999999999 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934076+0000", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019415319999999999 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934172+0000", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5725999999999998e-05 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934228+0000", 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6135999999999999e-05 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.236 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1067 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.2968870219999999, 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012883237000000001, 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1067 2024-09-11T05:51:21.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921720+0000", 2024-09-11T05:51:21.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4799999999999997e-07 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921725+0000", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7450000000000003e-06 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921734+0000", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3079999999999997e-06 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921743+0000", 2024-09-11T05:51:21.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5609999999999999e-06 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927241+0000", 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054980680000000001 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927487+0000", 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000245758 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.928564+0000", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010776990000000001 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933718+0000", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051533330000000004 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934356+0000", 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063792199999999997 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934504+0000", 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014797200000000001 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934603+0000", 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9023000000000004e-05 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1068 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.242 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.296804624, 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015284591, 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1068 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0500000000000003e-07 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921807+0000", 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9919999999999998e-06 2024-09-11T05:51:21.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921816+0000", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0999999999999993e-06 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921824+0000", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.255e-06 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929887+0000", 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080631929999999997 2024-09-11T05:51:21.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.930128+0000", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241318 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.931196+0000", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001067709 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934254+0000", 2024-09-11T05:51:21.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030585059999999999 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.936944+0000", 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002689126 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937038+0000", 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4772999999999995e-05 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937086+0000", 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8013999999999998e-05 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1069 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.2967398540000001, 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017393289999999999, 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1069 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921872+0000", 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6480000000000002e-06 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921880+0000", 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9850000000000006e-06 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921887+0000", 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932217+0000", 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010329680000000001 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932347+0000", 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129701 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932757+0000", 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000409904 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934275+0000", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015181579999999999 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939164+0000", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048894680000000001 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939216+0000", 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1721999999999997e-05 2024-09-11T05:51:21.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939260+0000", 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3782000000000002e-05 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1132 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.1844918779999993, 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011988883000000001, 2024-09-11T05:51:21.254 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1132 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-07 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034119+0000", 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1919999999999998e-06 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034132+0000", 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3036e-05 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034145+0000", 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2849e-05 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038262+0000", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041172459999999998 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038341+0000", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8678999999999997e-05 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038722+0000", 2024-09-11T05:51:21.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038121999999999999 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043113+0000", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004391112 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045962+0000", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028480300000000001 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046045+0000", 2024-09-11T05:51:21.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3294999999999994e-05 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046103+0000", 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8597999999999997e-05 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1133 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.1844464279999993, 2024-09-11T05:51:21.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012144546000000001, 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1133 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034161+0000", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.02e-07 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034166+0000", 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3090000000000003e-06 2024-09-11T05:51:21.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034174+0000", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034180+0000", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0100000000000001e-06 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039023+0000", 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048429850000000002 2024-09-11T05:51:21.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039070+0000", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7558000000000001e-05 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039252+0000", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000182051 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043138+0000", 2024-09-11T05:51:21.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003885234 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046192+0000", 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030545910000000002 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046257+0000", 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4529000000000005e-05 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046305+0000", 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7846000000000002e-05 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1351 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.5690836390000005, 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012039169000000001, 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.266 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1351 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4200000000000001e-07 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649526+0000", 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2389999999999998e-06 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649535+0000", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2279999999999998e-06 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649540+0000", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3759999999999999e-06 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657820+0000", 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082799009999999992 2024-09-11T05:51:21.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657923+0000", 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103447 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658388+0000", 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046494 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659901+0000", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015124719999999999 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661457+0000", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001555842 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661517+0000", 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9917999999999998e-05 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661562+0000", 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5364000000000001e-05 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1352 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.5690595409999997, 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014094584, 2024-09-11T05:51:21.272 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1352 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7099999999999997e-07 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649550+0000", 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6280000000000001e-06 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649556+0000", 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.246e-06 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649561+0000", 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6630000000000001e-06 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658923+0000", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093620470000000001 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659009+0000", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5964000000000006e-05 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659436+0000", 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042755199999999999 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660650+0000", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001213385 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663539+0000", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002889165 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663595+0000", 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6165999999999997e-05 2024-09-11T05:51:21.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663641+0000", 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6397000000000003e-05 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10124 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3604480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785080+0000", 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43352649799999998, 2024-09-11T05:51:21.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011413972, 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10124 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785080+0000", 2024-09-11T05:51:21.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785080+0000", 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785080+0000", 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.58e-07 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785083+0000", 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1519999999999999e-06 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785089+0000", 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2400000000000004e-06 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785104+0000", 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4825e-05 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796365+0000", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01126128 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796433+0000", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190999999999998e-05 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796494+0000", 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0825999999999997e-05 2024-09-11T05:51:21.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10125 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3670016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785117+0000", 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43348912899999997, 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011538636, 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10125 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785117+0000", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785117+0000", 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785118+0000", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2399999999999999e-07 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785120+0000", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4739999999999999e-06 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785128+0000", 2024-09-11T05:51:21.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8930000000000005e-06 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785133+0000", 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.972e-06 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796521+0000", 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011387599999999999 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796593+0000", 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2813999999999998e-05 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796656+0000", 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2459000000000006e-05 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10126 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3735552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785139+0000", 2024-09-11T05:51:21.287 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43346707499999998, 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011682857, 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10126 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785139+0000", 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785139+0000", 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785140+0000", 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7700000000000005e-07 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785142+0000", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.846e-06 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785148+0000", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1129999999999997e-06 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785152+0000", 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6720000000000003e-06 2024-09-11T05:51:21.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796685+0000", 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011532686 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796756+0000", 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0634000000000002e-05 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796822+0000", 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6428999999999997e-05 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10127 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.433447994, 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011864169000000001, 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10127 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785159+0000", 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0400000000000002e-07 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785161+0000", 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9400000000000001e-06 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785167+0000", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0179999999999998e-06 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785171+0000", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4649999999999996e-06 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796850+0000", 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011679172 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796923+0000", 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2836000000000005e-05 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797023+0000", 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9333999999999995e-05 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10128 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3866624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43342914399999999, 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012011195000000001, 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10128 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785178+0000", 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7899999999999999e-07 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785185+0000", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1570000000000003e-06 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785191+0000", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489999999999997e-06 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785196+0000", 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8189999999999999e-06 2024-09-11T05:51:21.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797053+0000", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011856716999999999 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797126+0000", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3351999999999998e-05 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797188+0000", 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2422000000000006e-05 2024-09-11T05:51:21.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10129 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43340386600000003, 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012703261, 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.301 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10129 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:21.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8000000000000001e-07 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785205+0000", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9769999999999999e-06 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785211+0000", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9630000000000003e-06 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785216+0000", 2024-09-11T05:51:21.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.656e-06 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797215+0000", 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011999697 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797819+0000", 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060352099999999998 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797906+0000", 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7066999999999998e-05 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10130 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43338468499999999, 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012870727, 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.305 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10130 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785224+0000", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.945e-06 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785231+0000", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4670000000000001e-06 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785237+0000", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1759999999999998e-06 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797938+0000", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012701097999999999 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798016+0000", 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7873000000000005e-05 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798092+0000", 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6749000000000007e-05 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10131 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.309 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.433363685, 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013020771, 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10131 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:21.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0200000000000003e-07 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785245+0000", 2024-09-11T05:51:21.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.838e-06 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785251+0000", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2090000000000002e-06 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785256+0000", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5029999999999999e-06 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798122+0000", 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012866803 2024-09-11T05:51:21.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798199+0000", 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6730000000000006e-05 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798263+0000", 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4286000000000001e-05 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10132 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.43332688899999999, 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013218944, 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10132 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9900000000000001e-07 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785283+0000", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8389999999999998e-06 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785297+0000", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4245e-05 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785302+0000", 2024-09-11T05:51:21.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3970000000000002e-06 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798345+0000", 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013042321000000001 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798421+0000", 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6666000000000001e-05 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798498+0000", 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6977000000000005e-05 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11483 3.17 3:ef8497a9:::rbd_data.1124518b2c12.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 3396905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.10813830200000001, 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014624382999999999, 2024-09-11T05:51:21.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11483 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110471+0000", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111721+0000", 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001250323 2024-09-11T05:51:21.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111747+0000", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6353999999999999e-05 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111760+0000", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2962000000000001e-05 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111830+0000", 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0217999999999998e-05 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111968+0000", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013758800000000001 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.112530+0000", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056166300000000001 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.121334+0000", 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088046630000000008 2024-09-11T05:51:21.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.124955+0000", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036208249999999998 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.125043+0000", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8189000000000006e-05 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.125092+0000", 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9191999999999999e-05 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.324 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:21.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1098 read 0x87ec154 thru 0x87fae7a (0xed27 bytes) 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1100 trunc from 0xbaa710a to 0xb1ff383 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1102 write 0xba3c3fa thru 0xba41c2d (0x5834 bytes) 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1103 write 0x89a2ff thru 0x8a76ad (0xd3af bytes) 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1104 write 0x6af066c thru 0x6afa0ab (0x9a40 bytes) 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1107 read 0x6bc812b thru 0x6bd14ae (0x9384 bytes) 2024-09-11T05:51:21.326 INFO:teuthology.orchestra.run.smithi165.stdout:1108 trunc from 0xba41c2e to 0x8420c4f 2024-09-11T05:51:21.327 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:21.326+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:21.334 INFO:teuthology.orchestra.run.smithi165.stdout:1110 trunc from 0x8420c4f to 0x2a2d10b 2024-09-11T05:51:21.368 INFO:teuthology.orchestra.run.smithi165.stdout:1111 write 0x43fb3d0 thru 0x4405fd2 (0xac03 bytes) 2024-09-11T05:51:21.389 INFO:teuthology.orchestra.run.smithi165.stdout:1112 write 0x6c47242 thru 0x6c49142 (0x1f01 bytes) 2024-09-11T05:51:21.412 INFO:teuthology.orchestra.run.smithi165.stdout:1113 write 0xae8f6d6 thru 0xae9f17f (0xfaaa bytes) 2024-09-11T05:51:21.426 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:21.426+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:21.433 INFO:teuthology.orchestra.run.smithi165.stdout:1114 writesame 0xbcdbbd5 thru 0xbce2dcc data_size 0x1c(0x71f8 bytes) 2024-09-11T05:51:21.451 INFO:teuthology.orchestra.run.smithi165.stdout:1115 writesame 0xd6da73f thru 0xd6e99f2 data_size 0x62(0xf2b4 bytes) 2024-09-11T05:51:21.451 INFO:teuthology.orchestra.run.smithi165.stdout:1119 write 0x7e9d080 thru 0x7e9d72a (0x6ab bytes) 2024-09-11T05:51:21.454 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:21.454 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:21.454 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:21.454 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:21.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:21.471 INFO:teuthology.orchestra.run.smithi165.stdout:1120 writesame 0xd70e46f thru 0xd7146fd data_size 0x1(0x628f bytes) 2024-09-11T05:51:21.471 INFO:teuthology.orchestra.run.smithi165.stdout:1121 writesame 0xb7ee16d thru 0xb7ef45f data_size 0x31(0x12f3 bytes) 2024-09-11T05:51:21.476 INFO:teuthology.orchestra.run.smithi165.stdout:1122 read 0x8266edb thru 0x8271f4d (0xb073 bytes) 2024-09-11T05:51:21.476 INFO:teuthology.orchestra.run.smithi165.stdout:1123 write 0x40e3fd2 thru 0x40ee89f (0xa8ce bytes) 2024-09-11T05:51:21.493 INFO:teuthology.orchestra.run.smithi165.stdout:1124 trunc from 0xd7146fe to 0x5065dc0 2024-09-11T05:51:21.517 INFO:teuthology.orchestra.run.smithi165.stdout:1126 read 0xdfadd4 thru 0xdfdf97 (0x31c4 bytes) 2024-09-11T05:51:21.526 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:21.526+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:21.537 INFO:teuthology.orchestra.run.smithi165.stdout:1127 write 0xbb3621c thru 0xbb430af (0xce94 bytes) 2024-09-11T05:51:21.543 INFO:teuthology.orchestra.run.smithi165.stdout:1128 read 0x4bbd700 thru 0x4bc5ac4 (0x83c5 bytes) 2024-09-11T05:51:21.543 INFO:teuthology.orchestra.run.smithi165.stdout:1129 read 0x2749573 thru 0x2753328 (0x9db6 bytes) 2024-09-11T05:51:21.543 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:21.544 INFO:teuthology.orchestra.run.smithi165.stdout:1131 trunc from 0xbb430b0 to 0xc804011 2024-09-11T05:51:21.554 INFO:teuthology.orchestra.run.smithi165.stdout:1133 write 0x7334798 thru 0x7334f09 (0x772 bytes) 2024-09-11T05:51:21.559 INFO:teuthology.orchestra.run.smithi165.stdout:1134 writesame 0x8b6c4c5 thru 0x8b7ad5f data_size 0x17(0xe89b bytes) 2024-09-11T05:51:21.565 INFO:teuthology.orchestra.run.smithi165.stdout:1136 write 0x5034557 thru 0x504238f (0xde39 bytes) 2024-09-11T05:51:21.569 INFO:teuthology.orchestra.run.smithi165.stdout:1137 writesame 0x39a2cd0 thru 0x39ad1ac data_size 0x17(0xa4dd bytes) 2024-09-11T05:51:21.579 INFO:teuthology.orchestra.run.smithi165.stdout:1139 trunc from 0xc804011 to 0x86acf23 2024-09-11T05:51:21.593 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T05:51:21.599 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:21.599 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:21.599 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:21.599 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:21.599 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:21.616 INFO:teuthology.orchestra.run.smithi165.stdout:1140 trunc from 0x86acf23 to 0x67e44d4 2024-09-11T05:51:21.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:21.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T05:51:21.627 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:21.626+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:21.646 INFO:teuthology.orchestra.run.smithi165.stdout:1141 write 0xcac8c83 thru 0xcad3d7a (0xb0f8 bytes) 2024-09-11T05:51:21.659 INFO:teuthology.orchestra.run.smithi165.stdout:1142 trunc from 0xcad3d7b to 0x7779e1c 2024-09-11T05:51:21.680 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:21.681 INFO:teuthology.orchestra.run.smithi165.stdout:1145 punch from 0x30d3d2d to 0x30db859, (0x7b2c bytes) 2024-09-11T05:51:21.698 INFO:teuthology.orchestra.run.smithi165.stdout:1146 write 0xc4042f0 thru 0xc404c40 (0x951 bytes) 2024-09-11T05:51:21.705 INFO:teuthology.orchestra.run.smithi165.stdout:1147 read 0xb960c5b thru 0xb96a661 (0x9a07 bytes) 2024-09-11T05:51:21.705 INFO:teuthology.orchestra.run.smithi165.stdout:1148 write 0x10f5fbb thru 0x11047ff (0xe845 bytes) 2024-09-11T05:51:21.728 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:21.730+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:21.742 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:21.742 INFO:teuthology.orchestra.run.smithi165.stdout:1150 read 0x441e998 thru 0x442ca0a (0xe073 bytes) 2024-09-11T05:51:21.743 INFO:teuthology.orchestra.run.smithi165.stdout:1151 writesame 0xa5e037a thru 0xa5ed7aa data_size 0x13(0xd431 bytes) 2024-09-11T05:51:21.750 INFO:teuthology.orchestra.run.smithi165.stdout:1152 write 0x9ecb86 thru 0x9fb0dc (0xe557 bytes) 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:953 3.19 3:9b18f687:::rbd_header.112475afb12f:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.1512469339999996, 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069395289999999998, 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.763 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 953 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611791+0000", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611793+0000", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611810+0000", 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6133999999999998e-05 2024-09-11T05:51:21.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611844+0000", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4403000000000002e-05 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611861+0000", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7246999999999999e-05 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.611991+0000", 2024-09-11T05:51:21.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012932899999999999 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.612003+0000", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2106000000000001e-05 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616192+0000", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041892049999999997 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616412+0000", 2024-09-11T05:51:21.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021982100000000001 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.616807+0000", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039489700000000001 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618206+0000", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013989510000000001 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618641+0000", 2024-09-11T05:51:21.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043535200000000002 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618695+0000", 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3740999999999999e-05 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.618730+0000", 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6023 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5115563889999999, 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069450629999999996, 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6023 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251481+0000", 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251482+0000", 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7800000000000001e-07 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251486+0000", 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.031e-06 2024-09-11T05:51:21.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251496+0000", 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0326000000000001e-05 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.251507+0000", 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-05 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.254914+0000", 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034066790000000001 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.255004+0000", 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0777000000000002e-05 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.255303+0000", 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029899400000000002 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.256292+0000", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009887889999999999 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258371+0000", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020787620000000001 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258403+0000", 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2110999999999998e-05 2024-09-11T05:51:21.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.258426+0000", 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2976000000000002e-05 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6094 3.4 3:25de1119:::rbd_data.11249b359231.0000000000000004:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4005698620000002, 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011224494999999999, 2024-09-11T05:51:21.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6094 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362469+0000", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362477+0000", 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:51:21.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362496+0000", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8598999999999999e-05 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362506+0000", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0336999999999999e-05 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362576+0000", 2024-09-11T05:51:21.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0401000000000006e-05 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362671+0000", 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4110999999999994e-05 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364415+0000", 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017441049999999999 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373558+0000", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091431189999999999 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373596+0000", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.803e-05 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373637+0000", 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0706e-05 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373692+0000", 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5531000000000001e-05 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6225 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1365720589999997, 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068693469999999996, 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6225 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626465+0000", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626466+0000", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626470+0000", 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3950000000000003e-06 2024-09-11T05:51:21.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626481+0000", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0178e-05 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.626488+0000", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.807e-06 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.630692+0000", 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042031359999999997 2024-09-11T05:51:21.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.630878+0000", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000186775 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.631345+0000", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046681899999999999 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.632338+0000", 2024-09-11T05:51:21.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099283500000000007 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633182+0000", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084431000000000003 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633267+0000", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4973000000000001e-05 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.633335+0000", 2024-09-11T05:51:21.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7519000000000002e-05 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6291 3.0 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2719744~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0406855669999997, 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072388169999999998, 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6291 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722352+0000", 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722354+0000", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.81e-06 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722369+0000", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5140000000000001e-05 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722389+0000", 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0594e-05 2024-09-11T05:51:21.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722399+0000", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6239999999999997e-06 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722501+0000", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101773 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722572+0000", 2024-09-11T05:51:21.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1155000000000006e-05 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722694+0000", 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121977 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.724008+0000", 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013144210000000001 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729476+0000", 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054675929999999998 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729538+0000", 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2123000000000001e-05 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729591+0000", 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2607e-05 2024-09-11T05:51:21.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6289 3.1a e33/23 3:5e4922eb:::rbd_data.11249b359231.000000000000001e:head v 33'20, mlcod=33'18)", 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0402184910000001, 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067092610000000002, 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722819+0000", 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722820+0000", 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2399999999999997e-07 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722835+0000", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4761e-05 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722847+0000", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2159e-05 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722854+0000", 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.481e-06 2024-09-11T05:51:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722940+0000", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5834999999999994e-05 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.722960+0000", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0171000000000001e-05 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729492+0000", 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065318980000000004 2024-09-11T05:51:21.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.729528+0000", 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6131999999999999e-05 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6339 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.9696937999999999, 2024-09-11T05:51:21.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073583650000000004, 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6339 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793344+0000", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.58e-07 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793347+0000", 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9529999999999999e-06 2024-09-11T05:51:21.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793354+0000", 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7870000000000004e-06 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.793359+0000", 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.339e-06 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796483+0000", 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031241120000000001 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796574+0000", 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0923999999999999e-05 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.796909+0000", 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033517300000000002 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.798000+0000", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001090382 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800606+0000", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026059970000000001 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800662+0000", 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6654999999999999e-05 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.800702+0000", 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9585000000000001e-05 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6690 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4537139200000002, 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067312420000000001, 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.804 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6690 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309324+0000", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6199999999999996e-07 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309328+0000", 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0960000000000003e-06 2024-09-11T05:51:21.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309338+0000", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2380000000000003e-06 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309344+0000", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1739999999999997e-06 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310323+0000", 2024-09-11T05:51:21.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00097914300000000003 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310421+0000", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7985000000000005e-05 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.310773+0000", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000351629 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315744+0000", 2024-09-11T05:51:21.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049715580000000001 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315787+0000", 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2413000000000002e-05 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.315937+0000", 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000150292 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316055+0000", 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117852 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6691 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4536525400000002, 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070656690000000001, 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6691 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309385+0000", 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309386+0000", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9800000000000003e-07 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309389+0000", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.614e-06 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309398+0000", 2024-09-11T05:51:21.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5799999999999992e-06 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309404+0000", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7180000000000001e-06 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.311472+0000", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002068482 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.311638+0000", 2024-09-11T05:51:21.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016590999999999999 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.312162+0000", 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052443200000000002 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316079+0000", 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039164250000000003 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316194+0000", 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115456 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316346+0000", 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151955 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316451+0000", 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104299 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6692 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4536104119999997, 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072663550000000004, 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6692 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.816 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309427+0000", 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309428+0000", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309431+0000", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4079999999999998e-06 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309438+0000", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.25e-06 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.309444+0000", 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6319999999999996e-06 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.312810+0000", 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003366047 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.313079+0000", 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026871099999999998 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.314118+0000", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001039119 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316483+0000", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002365481 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316579+0000", 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5902000000000004e-05 2024-09-11T05:51:21.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316650+0000", 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1020999999999996e-05 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.316693+0000", 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3167999999999997e-05 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6741 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3713894910000004, 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074311999999999998, 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6741 2024-09-11T05:51:21.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391648+0000", 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391649+0000", 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8199999999999999e-07 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391653+0000", 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1169999999999997e-06 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391662+0000", 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3910000000000004e-06 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391669+0000", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5189999999999998e-06 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.393939+0000", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022699920000000002 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394009+0000", 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0530000000000004e-05 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394218+0000", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020887100000000001 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398527+0000", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043089260000000002 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398947+0000", 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042018900000000001 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399034+0000", 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6650999999999994e-05 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399079+0000", 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5232000000000003e-05 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6742 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3713587250000003, 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075653420000000001, 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6742 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:21.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391679+0000", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5799999999999999e-07 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391683+0000", 2024-09-11T05:51:21.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.087e-06 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391692+0000", 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.507e-06 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391699+0000", 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1890000000000001e-06 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394755+0000", 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030557620000000001 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.394808+0000", 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2694000000000001e-05 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395007+0000", 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001998 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.398632+0000", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003624403 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399147+0000", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051573700000000005 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399203+0000", 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5464999999999998e-05 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399244+0000", 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1140000000000003e-05 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6743 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3713293200000001, 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077061630000000002, 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.833 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6743 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391708+0000", 2024-09-11T05:51:21.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391709+0000", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7700000000000005e-07 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391712+0000", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5099999999999999e-06 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391721+0000", 2024-09-11T05:51:21.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4530000000000002e-06 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.391727+0000", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4540000000000004e-06 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395418+0000", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036905359999999999 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.395574+0000", 2024-09-11T05:51:21.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015623899999999999 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.396102+0000", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000528135 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399304+0000", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032018469999999999 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399323+0000", 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9021999999999998e-05 2024-09-11T05:51:21.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399366+0000", 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3433999999999998e-05 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.399414+0000", 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8056000000000002e-05 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.838 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6779 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3143387200000003, 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071484610000000001, 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6779 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448699+0000", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448700+0000", 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.33e-07 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448706+0000", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2339999999999999e-06 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448717+0000", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0767e-05 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448723+0000", 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1460000000000001e-06 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.449522+0000", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000798696 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.449659+0000", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013719900000000001 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.450426+0000", 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076737400000000003 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.452781+0000", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023545739999999999 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455767+0000", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029862320000000001 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455816+0000", 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8646000000000001e-05 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455847+0000", 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1659999999999998e-05 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6781 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3143036229999998, 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007261254, 2024-09-11T05:51:21.844 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6781 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448734+0000", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448735+0000", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.16e-07 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448739+0000", 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7840000000000001e-06 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448747+0000", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2609999999999993e-06 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.448753+0000", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0360000000000003e-06 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451009+0000", 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022562770000000001 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451100+0000", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0957000000000003e-05 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.451485+0000", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038498900000000002 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455598+0000", 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041129440000000003 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455899+0000", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030118499999999998 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455956+0000", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6801e-05 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.455995+0000", 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9103999999999997e-05 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6803 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.475662+0000", 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.287375237, 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083572679999999993, 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6803 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475662+0000", 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475662+0000", 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475663+0000", 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475665+0000", 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5809999999999999e-06 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475671+0000", 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.475675+0000", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3220000000000003e-06 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.477590+0000", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019149709999999999 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.477687+0000", 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7187000000000003e-05 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.478073+0000", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038583699999999999 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.481002+0000", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029295340000000001 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.483890+0000", 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028872680000000001 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.483962+0000", 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1800999999999999e-05 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.484019+0000", 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7920000000000001e-05 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6848 3.19 3:9ed2d6d7:::rbd_object_map.11249b359231:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.572971+0000", 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.1900662200000003, 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077956270000000003, 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6848 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.572971+0000", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.572971+0000", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.572974+0000", 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7460000000000001e-06 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.573031+0000", 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6722999999999999e-05 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.573076+0000", 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5056999999999999e-05 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.573099+0000", 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2889000000000001e-05 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.573253+0000", 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154556 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.573486+0000", 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023307800000000001 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574254+0000", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076772099999999996 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580270+0000", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060163660000000004 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580610+0000", 2024-09-11T05:51:21.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033918199999999999 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580700+0000", 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0779000000000006e-05 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580767+0000", 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6530000000000002e-05 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11133 3.0 3:061d780f:::rbd_data.1124518b2c12.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.73450473800000005, 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03149018, 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11133 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.862 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028534+0000", 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7850000000000001e-06 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028542+0000", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8879999999999997e-06 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028559+0000", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6603999999999999e-05 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028567+0000", 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9019999999999999e-06 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028635+0000", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8046999999999994e-05 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028805+0000", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016979000000000001 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.033986+0000", 2024-09-11T05:51:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051812589999999997 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059864+0000", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025877539000000001 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059924+0000", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0327e-05 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059971+0000", 2024-09-11T05:51:21.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7239e-05 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060023+0000", 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1799999999999999e-05 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:11133 3.0 e36/23 ondisk, result = 0)", 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.71447176199999995, 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011468028, 2024-09-11T05:51:21.867 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048567+0000", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.19e-06 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048573+0000", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9290000000000003e-06 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048591+0000", 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8179999999999999e-05 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048601+0000", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8300000000000008e-06 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059910+0000", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011309476000000001 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059919+0000", 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9660000000000002e-06 2024-09-11T05:51:21.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060034+0000", 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011445700000000001 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:11636 3.1a e36/23 3:5ee2e31d:::rbd_data.1124518b2c12.0000000000000001:head v 36'46, mlcod=35'43)", 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.224013+0000", 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.53902466900000001, 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0092936679999999997, 2024-09-11T05:51:21.871 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.224013+0000", 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.224013+0000", 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.224017+0000", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2899999999999996e-06 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.227452+0000", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034345140000000001 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.227481+0000", 2024-09-11T05:51:21.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9164999999999999e-05 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.227496+0000", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5282999999999999e-05 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.227594+0000", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8005999999999996e-05 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.227624+0000", 2024-09-11T05:51:21.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0395e-05 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.233225+0000", 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056009579999999996 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.233306+0000", 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1057000000000007e-05 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:21.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:21.876 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:21.876 INFO:teuthology.orchestra.run.smithi165.stdout:1153 write 0xcb2587f thru 0xcb342a7 (0xea29 bytes) 2024-09-11T05:51:21.877 INFO:teuthology.orchestra.run.smithi165.stdout:1154 read 0x6ee68fc thru 0x6eed5df (0x6ce4 bytes) 2024-09-11T05:51:21.877 INFO:teuthology.orchestra.run.smithi165.stdout:1155 trunc from 0xcb342a8 to 0xc3b4ebc 2024-09-11T05:51:21.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1159 punch from 0xb29c7ff to 0xb2ab2a1, (0xeaa2 bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1160 trunc from 0xc3b4ebc to 0x8044e25 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1162 write 0xcbfe115 thru 0xcc0c2e9 (0xe1d5 bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1164 write 0x23185b7 thru 0x2325e34 (0xd87e bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1165 punch from 0x60944b8 to 0x6095b43, (0x168b bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1166 read 0x2ad6f14 thru 0x2ad8b59 (0x1c46 bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1168 read 0x96af030 thru 0x96b7dbe (0x8d8f bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1170 read 0x38b5814 thru 0x38c3fde (0xe7cb bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1171 write 0x3b4c303 thru 0x3b4f60c (0x330a bytes) 2024-09-11T05:51:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:1172 trunc from 0xcc0c2ea to 0xeadd0c3 2024-09-11T05:51:21.879 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:21.874+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:21.885 INFO:teuthology.orchestra.run.smithi165.stdout:1173 trunc from 0xeadd0c3 to 0x92e5054 2024-09-11T05:51:21.907 INFO:teuthology.orchestra.run.smithi165.stdout:1174 writesame 0x407a875 thru 0x40817a9 data_size 0x31(0x6f35 bytes) 2024-09-11T05:51:21.910 INFO:teuthology.orchestra.run.smithi165.stdout:1176 writesame 0x1ad32d1 thru 0x1ad38e3 data_size 0x1(0x613 bytes) 2024-09-11T05:51:21.914 INFO:teuthology.orchestra.run.smithi165.stdout:1177 punch from 0x6966371 to 0x696eff2, (0x8c81 bytes) 2024-09-11T05:51:21.927 INFO:teuthology.orchestra.run.smithi165.stdout:1178 write 0x9c240b5 thru 0x9c2f2c0 (0xb20c bytes) 2024-09-11T05:51:21.934 INFO:teuthology.orchestra.run.smithi165.stdout:1180 read 0x95e0cc1 thru 0x95e6c0e (0x5f4e bytes) 2024-09-11T05:51:21.934 INFO:teuthology.orchestra.run.smithi165.stdout:1181 write 0x2fa17f5 thru 0x2faa358 (0x8b64 bytes) 2024-09-11T05:51:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:1185 write 0xc4cf2d thru 0xc4ed1e (0x1df2 bytes) 2024-09-11T05:51:21.959 INFO:teuthology.orchestra.run.smithi165.stdout:1187 write 0xb349d95 thru 0xb359782 (0xf9ee bytes) 2024-09-11T05:51:21.978 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:21.974+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:21.978 INFO:teuthology.orchestra.run.smithi165.stdout:1190 write 0xce1f1bb thru 0xce291ed (0xa033 bytes) 2024-09-11T05:51:21.998 INFO:teuthology.orchestra.run.smithi165.stdout:1191 trunc from 0xce291ee to 0x2235537 2024-09-11T05:51:22.007 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:22.008 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:22.008 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:22.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:22.063 INFO:teuthology.orchestra.run.smithi165.stdout:1193 write 0xbe4032e thru 0xbe439ff (0x36d2 bytes) 2024-09-11T05:51:22.069 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.069 INFO:teuthology.orchestra.run.smithi165.stdout:1196 write 0x1560cf3 thru 0x1562de3 (0x20f1 bytes) 2024-09-11T05:51:22.079 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:22.078+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:22.094 INFO:teuthology.orchestra.run.smithi165.stdout:1197 writesame 0x36cc2b thru 0x373d1f data_size 0x15(0x70f5 bytes) 2024-09-11T05:51:22.099 INFO:teuthology.orchestra.run.smithi165.stdout:1200 read 0x4444c68 thru 0x44545fc (0xf995 bytes) 2024-09-11T05:51:22.099 INFO:teuthology.orchestra.run.smithi165.stdout:1201 punch from 0x72111cb to 0x7212843, (0x1678 bytes) 2024-09-11T05:51:22.099 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.099 INFO:teuthology.orchestra.run.smithi165.stdout:1203 writesame 0x7cd99ec thru 0x7ce87c8 data_size 0x1(0xeddd bytes) 2024-09-11T05:51:22.179 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:22.178+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:22.181 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.181 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:22.181 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:22.181 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:22.181 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:22.196 INFO:teuthology.orchestra.run.smithi165.stdout:1205 trunc from 0xbe43a00 to 0xb95e4d 2024-09-11T05:51:22.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:22.238 INFO:teuthology.orchestra.run.smithi165.stdout:1206 trunc from 0xb95e4d to 0x21cd088 2024-09-11T05:51:22.250 INFO:teuthology.orchestra.run.smithi165.stdout:1211 read 0x2080a8d thru 0x208aea6 (0xa41a bytes) 2024-09-11T05:51:22.251 INFO:teuthology.orchestra.run.smithi165.stdout:1212 read 0x12b0a16 thru 0x12b6520 (0x5b0b bytes) 2024-09-11T05:51:22.252 INFO:teuthology.orchestra.run.smithi165.stdout:1214 trunc from 0x21cd088 to 0x291c9c7 2024-09-11T05:51:22.264 INFO:teuthology.orchestra.run.smithi165.stdout:1215 punch from 0xdeafcc to 0xdef976, (0x49aa bytes) 2024-09-11T05:51:22.280 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:22.278+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:22.281 INFO:teuthology.orchestra.run.smithi165.stdout:1218 writesame 0x613002d thru 0x6130a25 data_size 0x17(0x9f9 bytes) 2024-09-11T05:51:22.281 INFO:teuthology.orchestra.run.smithi165.stdout:1219 read 0x2a5da60 thru 0x2a5fb91 (0x2132 bytes) 2024-09-11T05:51:22.281 INFO:teuthology.orchestra.run.smithi165.stdout:1220 read 0x3dbc8b8 thru 0x3dc8d5a (0xc4a3 bytes) 2024-09-11T05:51:22.303 INFO:teuthology.orchestra.run.smithi165.stdout:1221 writesame 0xd3b0f37 thru 0xd3bcdfd data_size 0x1(0xbec7 bytes) 2024-09-11T05:51:22.303 INFO:teuthology.orchestra.run.smithi165.stdout:1222 write 0x598fa6f thru 0x5999308 (0x989a bytes) 2024-09-11T05:51:22.307 INFO:teuthology.orchestra.run.smithi165.stdout:1223 read 0x4a76918 thru 0x4a84f1c (0xe605 bytes) 2024-09-11T05:51:22.308 INFO:teuthology.orchestra.run.smithi165.stdout:1225 writesame 0x87a276a thru 0x87a3999 data_size 0x18(0x1230 bytes) 2024-09-11T05:51:22.315 INFO:teuthology.orchestra.run.smithi165.stdout:1226 punch from 0x7e77313 to 0x7e78675, (0x1362 bytes) 2024-09-11T05:51:22.315 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.315 INFO:teuthology.orchestra.run.smithi165.stdout:1228 read 0x9c2f405 thru 0x9c35d55 (0x6951 bytes) 2024-09-11T05:51:22.315 INFO:teuthology.orchestra.run.smithi165.stdout:1229 write 0xdc1aa thru 0xdf4a6 (0x32fd bytes) 2024-09-11T05:51:22.319 INFO:teuthology.orchestra.run.smithi165.stdout:1231 read 0x8e40d8a thru 0x8e47a3c (0x6cb3 bytes) 2024-09-11T05:51:22.319 INFO:teuthology.orchestra.run.smithi165.stdout:1232 read 0x6856d61 thru 0x6865453 (0xe6f3 bytes) 2024-09-11T05:51:22.319 INFO:teuthology.orchestra.run.smithi165.stdout:1233 writesame 0x867476b thru 0x8675496 data_size 0x6(0xd2c bytes) 2024-09-11T05:51:22.323 INFO:teuthology.orchestra.run.smithi165.stdout:1234 writesame 0x804d6d5 thru 0x8051809 data_size 0x1(0x4135 bytes) 2024-09-11T05:51:22.323 INFO:teuthology.orchestra.run.smithi165.stdout:1235 read 0xa96bc24 thru 0xa96f519 (0x38f6 bytes) 2024-09-11T05:51:22.324 INFO:teuthology.orchestra.run.smithi165.stdout:1236 writesame 0xbfd66ce thru 0xbfdd62f data_size 0x35(0x6f62 bytes) 2024-09-11T05:51:22.331 INFO:teuthology.orchestra.run.smithi165.stdout:1237 read 0x5b18a40 thru 0x5b1f59c (0x6b5d bytes) 2024-09-11T05:51:22.345 INFO:teuthology.orchestra.run.smithi165.stdout:1239 writesame 0xe0b9670 thru 0xe0bb2ba data_size 0x1(0x1c4b bytes) 2024-09-11T05:51:22.345 INFO:teuthology.orchestra.run.smithi165.stdout:1240 punch from 0x178e0f5 to 0x1797527, (0x9432 bytes) 2024-09-11T05:51:22.346 INFO:teuthology.orchestra.run.smithi165.stdout:1241 punch from 0xb759d56 to 0xb763b1c, (0x9dc6 bytes) 2024-09-11T05:51:22.346 INFO:teuthology.orchestra.run.smithi165.stdout:1243 read 0x537d96f thru 0x53881e6 (0xa878 bytes) 2024-09-11T05:51:22.346 INFO:teuthology.orchestra.run.smithi165.stdout:1244 punch from 0xa922308 to 0xa931c6c, (0xf964 bytes) 2024-09-11T05:51:22.346 INFO:teuthology.orchestra.run.smithi165.stdout:1245 write 0x3d2f1c3 thru 0x3d30128 (0xf66 bytes) 2024-09-11T05:51:22.353 INFO:teuthology.orchestra.run.smithi165.stdout:1246 write 0x8fda957 thru 0x8fdf9c7 (0x5071 bytes) 2024-09-11T05:51:22.360 INFO:teuthology.orchestra.run.smithi165.stdout:1247 write 0xce3a14 thru 0xce8ac4 (0x50b1 bytes) 2024-09-11T05:51:22.366 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.366 INFO:teuthology.orchestra.run.smithi165.stdout:1250 read 0x3ca5a35 thru 0x3cad341 (0x790d bytes) 2024-09-11T05:51:22.368 INFO:teuthology.orchestra.run.smithi165.stdout:1252 read 0x6eecfdd thru 0x6ef2366 (0x538a bytes) 2024-09-11T05:51:22.368 INFO:teuthology.orchestra.run.smithi165.stdout:1254 writesame 0x8b34e6b thru 0x8b3b5f9 data_size 0x1(0x678f bytes) 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4239.0:1 2.3 e20/19 2:c4c92e5a:::rbd_trash:head v 20'1, mlcod=0'0)", 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.208524140000002, 2024-09-11T05:51:22.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017074670000000001, 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158544+0000", 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158545+0000", 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4e-07 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158565+0000", 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0299999999999999e-05 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158591+0000", 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6146e-05 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158613+0000", 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1647000000000001e-05 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158753+0000", 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014034200000000001 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.158836+0000", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3249000000000002e-05 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160169+0000", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013321769999999999 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:50:56.160251+0000", 2024-09-11T05:51:22.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2766e-05 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.1551310969999999, 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:22.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6053 3.0 e33/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0636667309999996, 2024-09-11T05:51:22.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019874799999999998, 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:22.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303403+0000", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.378e-06 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303414+0000", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.155e-05 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303437+0000", 2024-09-11T05:51:22.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2342000000000001e-05 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303446+0000", 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6760000000000002e-06 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303513+0000", 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6431000000000001e-05 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303536+0000", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3258999999999999e-05 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305349+0000", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018126640000000001 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305389+0000", 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0179999999999998e-05 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6060 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'29, mlcod=33'27)", 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0528204619999997, 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024674810000000001, 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:22.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4099999999999997e-07 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314255+0000", 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314269+0000", 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4177e-05 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314277+0000", 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6279999999999996e-06 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314341+0000", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3539000000000003e-05 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314363+0000", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2272e-05 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316606+0000", 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002243629 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316715+0000", 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108576 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0521413659999999, 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.387 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.391 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6091 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0099814, 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019923990000000002, 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357088+0000", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3829999999999999e-06 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357127+0000", 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8819999999999998e-05 2024-09-11T05:51:22.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357144+0000", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7027000000000001e-05 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357153+0000", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5810000000000003e-06 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357213+0000", 2024-09-11T05:51:22.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9959e-05 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357231+0000", 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7983e-05 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359040+0000", 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018086090000000001 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359079+0000", 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9036999999999999e-05 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9575993299999999, 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.396 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:22.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6858520459999999, 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:22.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.404 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6343 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'23, mlcod=33'21)", 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5654371710000001, 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001815929, 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801642+0000", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1019000000000001e-05 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801673+0000", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1349000000000001e-05 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801694+0000", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1067999999999999e-05 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801706+0000", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1731e-05 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801771+0000", 2024-09-11T05:51:22.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4613e-05 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801789+0000", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8839e-05 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803393+0000", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016039299999999999 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803447+0000", 2024-09-11T05:51:22.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3380000000000001e-05 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5618330719999998, 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6354 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'25, mlcod=33'23)", 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5504598999999999, 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026120029999999999, 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.413 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816609+0000", 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:22.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816617+0000", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7680000000000008e-06 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816630+0000", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816637+0000", 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816690+0000", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2533e-05 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816706+0000", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6407000000000001e-05 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819170+0000", 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024645309999999998 2024-09-11T05:51:22.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819220+0000", 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9595999999999997e-05 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4868408960000004, 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:22.417 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:22.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:22.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.421 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6672 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'39, mlcod=33'37)", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0682872979999996, 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022554760000000002, 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298783+0000", 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.362e-06 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298845+0000", 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1544999999999994e-05 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298878+0000", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3642000000000001e-05 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298895+0000", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6750000000000001e-05 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299066+0000", 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017087600000000001 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299124+0000", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8213e-05 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.300980+0000", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00185586 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.301036+0000", 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6227999999999998e-05 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.425 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0128657759999999, 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:22.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:22.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:22.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:22.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6924 3.0 e34/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 34'25, mlcod=34'23)", 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.665635129, 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019233869999999999, 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2e-07 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701440+0000", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.314e-06 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701448+0000", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1939999999999998e-06 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701474+0000", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6336999999999999e-05 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701570+0000", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5503999999999999e-05 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701588+0000", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8510000000000001e-05 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703303+0000", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017153380000000001 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703356+0000", 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2769999999999998e-05 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.5202365120000003, 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.434 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:22.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:22.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:22.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7116 3.9 e34/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 34'35, mlcod=34'33)", 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3600085350000004, 2024-09-11T05:51:22.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00371405, 2024-09-11T05:51:22.439 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.439 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007061+0000", 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5090000000000001e-06 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007072+0000", 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1399e-05 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007100+0000", 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7531000000000001e-05 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007114+0000", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4378e-05 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007340+0000", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022568499999999999 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007387+0000", 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7194000000000002e-05 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010717+0000", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033296369999999999 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010773+0000", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6716999999999999e-05 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7119 3.4 e34/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 34'49, mlcod=34'47)", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:22.443 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.3588027570000003, 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034162110000000002, 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008267+0000", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.294e-06 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008276+0000", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5820000000000008e-06 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008290+0000", 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4005000000000001e-05 2024-09-11T05:51:22.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008298+0000", 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3189999999999995e-06 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008368+0000", 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0055999999999995e-05 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008396+0000", 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7563999999999999e-05 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011628+0000", 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032314969999999998 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011681+0000", 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3894000000000003e-05 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7194 3.4 e34/23 3:2779ff95:::rbd_data.11249b359231.000000000000001f:head v 34'50, mlcod=34'48)", 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.103297+0000", 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2637710469999996, 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001674785, 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103297+0000", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103297+0000", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103300+0000", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7470000000000001e-06 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103367+0000", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7223000000000003e-05 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103410+0000", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2769999999999999e-05 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103431+0000", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1682999999999999e-05 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103637+0000", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020582200000000001 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.103692+0000", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5325999999999998e-05 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.104864+0000", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011716579999999999 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.104972+0000", 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107556 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12191 3.0 e37/23 3:0506084d:::rbd_data.1124518b2c12.0000000000000006:head v 37'33, mlcod=36'31)", 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.943973+0000", 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.42309530899999997, 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001696005, 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.943973+0000", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.943973+0000", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.943973+0000", 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2099999999999998e-07 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.943996+0000", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2583e-05 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.944009+0000", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3216e-05 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.944022+0000", 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3098999999999999e-05 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.944154+0000", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132125 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.944194+0000", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9579000000000002e-05 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.945564+0000", 2024-09-11T05:51:22.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001370776 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.945669+0000", 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104306 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.456 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:22.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1255 read 0x3506eaa thru 0x35129e7 (0xbb3e bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1256 write 0x221e771 thru 0x2221a7f (0x330f bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1257 read 0x7ff7962 thru 0x7fff457 (0x7af6 bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1262 write 0xe0c27b9 thru 0xe0cec93 (0xc4db bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1264 write 0x9aa35e thru 0x9b61ab (0xbe4e bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1265 read 0x96f26c0 thru 0x97013c6 (0xed07 bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1266 read 0x10d5c2e thru 0x10e1f8c (0xc35f bytes) 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1267 trunc from 0xe0cec94 to 0xe0eae72 2024-09-11T05:51:22.460 INFO:teuthology.orchestra.run.smithi165.stdout:1268 read 0x4d1a77f thru 0x4d1b00f (0x891 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1269 punch from 0xa0ff677 to 0xa1044ab, (0x4e34 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1274 writesame 0x160835c thru 0x1611e1f data_size 0x46(0x9ac4 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1275 read 0x96c938d thru 0x96cfb74 (0x67e8 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1276 punch from 0x81c9376 to 0x81d7d5d, (0xe9e7 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1279 punch from 0x84369fb to 0x843bac7, (0x50cc bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1280 write 0x9f3c79f thru 0x9f45266 (0x8ac8 bytes) 2024-09-11T05:51:22.461 INFO:teuthology.orchestra.run.smithi165.stdout:1281 write 0x8caab0c thru 0x8cab7b7 (0xcac bytes) 2024-09-11T05:51:22.461 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:22.454+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:22.462 INFO:teuthology.orchestra.run.smithi165.stdout:1283 read 0xbce2f84 thru 0xbcf278f (0xf80c bytes) 2024-09-11T05:51:22.462 INFO:teuthology.orchestra.run.smithi165.stdout:1285 punch from 0xdb83545 to 0xdb86dba, (0x3875 bytes) 2024-09-11T05:51:22.462 INFO:teuthology.orchestra.run.smithi165.stdout:1286 read 0x78d18d9 thru 0x78d2348 (0xa70 bytes) 2024-09-11T05:51:22.462 INFO:teuthology.orchestra.run.smithi165.stdout:1287 writesame 0x9d67fe thru 0x9e3b09 data_size 0x1a(0xd30c bytes) 2024-09-11T05:51:22.467 INFO:teuthology.orchestra.run.smithi165.stdout:1288 write 0x548c581 thru 0x5492da2 (0x6822 bytes) 2024-09-11T05:51:22.473 INFO:teuthology.orchestra.run.smithi165.stdout:1289 write 0xafbdcec thru 0xafc90c5 (0xb3da bytes) 2024-09-11T05:51:22.476 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.476 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.476 INFO:teuthology.orchestra.run.smithi165.stdout:1292 read 0xac1996e thru 0xac2554e (0xbbe1 bytes) 2024-09-11T05:51:22.476 INFO:teuthology.orchestra.run.smithi165.stdout:1293 read 0xb0e4cbf thru 0xb0e8f1a (0x425c bytes) 2024-09-11T05:51:22.476 INFO:teuthology.orchestra.run.smithi165.stdout:1294 read 0x7291196 thru 0x7293596 (0x2401 bytes) 2024-09-11T05:51:22.498 INFO:teuthology.orchestra.run.smithi165.stdout:1295 trunc from 0xe0eae72 to 0xf0c1b7 2024-09-11T05:51:22.554 INFO:teuthology.orchestra.run.smithi165.stdout:1296 write 0x9228004 thru 0x922a29a (0x2297 bytes) 2024-09-11T05:51:22.558 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:22.554+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:22.561 INFO:teuthology.orchestra.run.smithi165.stdout:1297 read 0x250a298 thru 0x2519dd4 (0xfb3d bytes) 2024-09-11T05:51:22.561 INFO:teuthology.orchestra.run.smithi165.stdout:1298 read 0x139ed29 thru 0x13a39a4 (0x4c7c bytes) 2024-09-11T05:51:22.562 INFO:teuthology.orchestra.run.smithi165.stdout:1299 read 0x493c374 thru 0x4941347 (0x4fd4 bytes) 2024-09-11T05:51:22.562 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.562 INFO:teuthology.orchestra.run.smithi165.stdout:1303 punch from 0x8a0d4bf to 0x8a139c5, (0x6506 bytes) 2024-09-11T05:51:22.562 INFO:teuthology.orchestra.run.smithi165.stdout:1304 write 0x7d961fc thru 0x7d97289 (0x108e bytes) 2024-09-11T05:51:22.568 INFO:teuthology.orchestra.run.smithi165.stdout:1305 punch from 0x5f9efb7 to 0x5fa811f, (0x9168 bytes) 2024-09-11T05:51:22.568 INFO:teuthology.orchestra.run.smithi165.stdout:1308 write 0x2793a83 thru 0x2796c7f (0x31fd bytes) 2024-09-11T05:51:22.574 INFO:teuthology.orchestra.run.smithi165.stdout:1309 punch from 0x5badfa8 to 0x5bb0ee2, (0x2f3a bytes) 2024-09-11T05:51:22.574 INFO:teuthology.orchestra.run.smithi165.stdout:1310 read 0x2559344 thru 0x255b919 (0x25d6 bytes) 2024-09-11T05:51:22.574 INFO:teuthology.orchestra.run.smithi165.stdout:1311 read 0x6f4cd7c thru 0x6f4d2e4 (0x569 bytes) 2024-09-11T05:51:22.575 INFO:teuthology.orchestra.run.smithi165.stdout:1312 punch from 0x6b63a3 to 0x6b9c7c, (0x38d9 bytes) 2024-09-11T05:51:22.575 INFO:teuthology.orchestra.run.smithi165.stdout:1313 punch from 0x6d20b5b to 0x6d3060b, (0xfab0 bytes) 2024-09-11T05:51:22.575 INFO:teuthology.orchestra.run.smithi165.stdout:1314 read 0x621132c thru 0x621a6cc (0x93a1 bytes) 2024-09-11T05:51:22.575 INFO:teuthology.orchestra.run.smithi165.stdout:1315 write 0x13b80a2 thru 0x13c6a8c (0xe9eb bytes) 2024-09-11T05:51:22.585 INFO:teuthology.orchestra.run.smithi165.stdout:1316 trunc from 0x922a29b to 0xd94f0e1 2024-09-11T05:51:22.598 INFO:teuthology.orchestra.run.smithi165.stdout:1318 read 0x15ab572 thru 0x15b5999 (0xa428 bytes) 2024-09-11T05:51:22.599 INFO:teuthology.orchestra.run.smithi165.stdout:1320 write 0xcc7ca00 thru 0xcc8b01d (0xe61e bytes) 2024-09-11T05:51:22.599 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:22.600 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:22.600 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:22.606 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T05:51:22.615 INFO:teuthology.orchestra.run.smithi165.stdout:1321 read 0x1a4387a thru 0x1a4d40f (0x9b96 bytes) 2024-09-11T05:51:22.618 INFO:teuthology.orchestra.run.smithi165.stdout:1322 writesame 0x2562e81 thru 0x2562f19 data_size 0x3(0x99 bytes) 2024-09-11T05:51:22.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:22.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:51:22.624 INFO:teuthology.orchestra.run.smithi165.stdout:1323 write 0xb56fad4 thru 0xb57453d (0x4a6a bytes) 2024-09-11T05:51:22.630 INFO:teuthology.orchestra.run.smithi165.stdout:1324 read 0xaffebbb thru 0xb00a1f2 (0xb638 bytes) 2024-09-11T05:51:22.630 INFO:teuthology.orchestra.run.smithi165.stdout:1325 read 0xbbc39cc thru 0xbbd2c7b (0xf2b0 bytes) 2024-09-11T05:51:22.630 INFO:teuthology.orchestra.run.smithi165.stdout:1326 write 0x11c88a thru 0x11db39 (0x12b0 bytes) 2024-09-11T05:51:22.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T05:51:22.634 INFO:teuthology.orchestra.run.smithi165.stdout:1327 read 0x4e39110 thru 0x4e47778 (0xe669 bytes) 2024-09-11T05:51:22.634 INFO:teuthology.orchestra.run.smithi165.stdout:1329 write 0x64748f5 thru 0x6480f1c (0xc628 bytes) 2024-09-11T05:51:22.639 INFO:teuthology.orchestra.run.smithi165.stdout:1330 punch from 0xb554d66 to 0xb55b7bb, (0x6a55 bytes) 2024-09-11T05:51:22.639 INFO:teuthology.orchestra.run.smithi165.stdout:1331 punch from 0x1a4ed11 to 0x1a50277, (0x1566 bytes) 2024-09-11T05:51:22.648 INFO:teuthology.orchestra.run.smithi165.stdout:1332 trunc from 0xd94f0e1 to 0x83efced 2024-09-11T05:51:22.659 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:22.654+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:22.672 INFO:teuthology.orchestra.run.smithi165.stdout:1333 writesame 0xa30378 thru 0xa4036b data_size 0x2(0xfff4 bytes) 2024-09-11T05:51:22.695 INFO:teuthology.orchestra.run.smithi165.stdout:1334 read 0x11f787a thru 0x1202fbf (0xb746 bytes) 2024-09-11T05:51:22.696 INFO:teuthology.orchestra.run.smithi165.stdout:1335 trunc from 0x83efced to 0x8417776 2024-09-11T05:51:22.709 INFO:teuthology.orchestra.run.smithi165.stdout:1336 writesame 0x1d6fc5a thru 0x1d6fe3b data_size 0x1(0x1e2 bytes) 2024-09-11T05:51:22.711 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.732 INFO:teuthology.orchestra.run.smithi165.stdout:1340 writesame 0xd68abb1 thru 0xd68d562 data_size 0x9(0x29b2 bytes) 2024-09-11T05:51:22.745 INFO:teuthology.orchestra.run.smithi165.stdout:1342 writesame 0xdf89d23 thru 0xdf8bb71 data_size 0x1(0x1e4f bytes) 2024-09-11T05:51:22.745 INFO:teuthology.orchestra.run.smithi165.stdout:1343 write 0x42b3cee thru 0x42bead0 (0xade3 bytes) 2024-09-11T05:51:22.751 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.751 INFO:teuthology.orchestra.run.smithi165.stdout:1346 writesame 0x5cb29e3 thru 0x5cb8ec0 data_size 0x1(0x64de bytes) 2024-09-11T05:51:22.759 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:22.758+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:22.762 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.762 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:22.762 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:22.762 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:22.762 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:22.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:22.790 INFO:teuthology.orchestra.run.smithi165.stdout:1347 read 0xf27940 thru 0xf2f385 (0x7a46 bytes) 2024-09-11T05:51:22.790 INFO:teuthology.orchestra.run.smithi165.stdout:1348 write 0x63e4000 thru 0x63f1862 (0xd863 bytes) 2024-09-11T05:51:22.795 INFO:teuthology.orchestra.run.smithi165.stdout:1349 read 0xb03a4e9 thru 0xb043639 (0x9151 bytes) 2024-09-11T05:51:22.796 INFO:teuthology.orchestra.run.smithi165.stdout:1351 writesame 0xb413640 thru 0xb41e057 data_size 0x4(0xaa18 bytes) 2024-09-11T05:51:22.804 INFO:teuthology.orchestra.run.smithi165.stdout:1352 read 0x61e1dc8 thru 0x61ef183 (0xd3bc bytes) 2024-09-11T05:51:22.804 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.805 INFO:teuthology.orchestra.run.smithi165.stdout:1354 punch from 0x496227b to 0x4968c41, (0x69c6 bytes) 2024-09-11T05:51:22.805 INFO:teuthology.orchestra.run.smithi165.stdout:1355 write 0xa67101b thru 0xa6764f7 (0x54dd bytes) 2024-09-11T05:51:22.813 INFO:teuthology.orchestra.run.smithi165.stdout:1356 trunc from 0xdf8bb72 to 0xc8b8800 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:1357 read 0xa11da87 thru 0xa11e802 (0xd7c bytes) 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:1358 read 0x325c18a thru 0x3262d8d (0x6c04 bytes) 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:1360 punch from 0x8f18373 to 0x8f1f73d, (0x73ca bytes) 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:1364 read 0x45eecd1 thru 0x45ef71b (0xa4b bytes) 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.828 INFO:teuthology.orchestra.run.smithi165.stdout:1366 punch from 0x3c5dcf9 to 0x3c61e17, (0x411e bytes) 2024-09-11T05:51:22.829 INFO:teuthology.orchestra.run.smithi165.stdout:1367 read 0x5fde5c7 thru 0x5fecfa7 (0xe9e1 bytes) 2024-09-11T05:51:22.829 INFO:teuthology.orchestra.run.smithi165.stdout:1368 write 0x6de938c thru 0x6def50c (0x6181 bytes) 2024-09-11T05:51:22.831 INFO:teuthology.orchestra.run.smithi165.stdout:1370 punch from 0x9e558d to 0x9f512e, (0xfba1 bytes) 2024-09-11T05:51:22.849 INFO:teuthology.orchestra.run.smithi165.stdout:1371 trunc from 0xc8b8800 to 0x236c87 2024-09-11T05:51:22.860 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:22.858+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:22.890 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:22.890 INFO:teuthology.orchestra.run.smithi165.stdout:1375 read 0x1f307d thru 0x1f5202 (0x2186 bytes) 2024-09-11T05:51:22.917 INFO:teuthology.orchestra.run.smithi165.stdout:1376 write 0xa425279 thru 0xa4297f9 (0x4581 bytes) 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:881 3.b e28/23 ondisk, result = 0)", 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.485875077999999, 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010956011, 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438671+0000", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438675+0000", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1910000000000002e-06 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438685+0000", 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0001e-05 2024-09-11T05:51:22.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438692+0000", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3830000000000003e-06 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449422+0000", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010729394 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449429+0000", 2024-09-11T05:51:22.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6599999999999995e-06 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449626+0000", 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019679800000000001 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:913 3.b 3:d210a54d:::rbd_data.11246a69a978.0000000000000032:head [set-alloc-hint object_size 4194304 write_size 4194304,write 870611~56132] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:11.521261+0000", 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.403284055, 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071823, 2024-09-11T05:51:22.929 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 913 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521261+0000", 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521261+0000", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521263+0000", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8160000000000001e-06 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521286+0000", 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3190999999999999e-05 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521303+0000", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7411999999999999e-05 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521311+0000", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9810000000000003e-06 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521370+0000", 2024-09-11T05:51:22.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8106999999999998e-05 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.521376+0000", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0159999999999997e-06 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.525444+0000", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00406807 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.525600+0000", 2024-09-11T05:51:22.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156212 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.525767+0000", 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016683 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.528116+0000", 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023489029999999998 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.528367+0000", 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025190999999999999 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.528407+0000", 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9780999999999997e-05 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.528443+0000", 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6071000000000001e-05 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1117 3.18 3:1e873f29:::rbd_data.112475afb12f.0000000000000039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.023990+0000", 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.9005547420000006, 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065041750000000001, 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1117 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.023990+0000", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.023990+0000", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.023991+0000", 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.92e-07 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.023998+0000", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3470000000000001e-06 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024011+0000", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2867999999999999e-05 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024019+0000", 2024-09-11T05:51:22.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8909999999999995e-06 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024440+0000", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042029099999999999 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024551+0000", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111088 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024702+0000", 2024-09-11T05:51:22.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015165299999999999 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027949+0000", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032467730000000001 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030436+0000", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024873389999999999 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030469+0000", 2024-09-11T05:51:22.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2379999999999998e-05 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030495+0000", 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5752999999999999e-05 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1120 3.18 3:1bce9b53:::rbd_data.112475afb12f.000000000000003c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.024036+0000", 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.9005094479999993, 2024-09-11T05:51:22.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065373430000000001, 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1120 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024036+0000", 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024036+0000", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024036+0000", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4399999999999999e-07 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024041+0000", 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1169999999999997e-06 2024-09-11T05:51:22.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024050+0000", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9720000000000004e-06 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.024058+0000", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3139999999999998e-06 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.025014+0000", 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095651199999999997 2024-09-11T05:51:22.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.025121+0000", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107001 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.025191+0000", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9547000000000003e-05 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.027952+0000", 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027613260000000001 2024-09-11T05:51:22.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030525+0000", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025723959999999998 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030551+0000", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6835999999999999e-05 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.030573+0000", 2024-09-11T05:51:22.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1577999999999999e-05 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1198 3.8 3:15f20ad5:::rbd_data.112475afb12f.00000000000000b6:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.7746450449999998, 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013528144000000001, 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1198 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149901+0000", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.172e-06 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149915+0000", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3801000000000001e-05 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149929+0000", 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3624e-05 2024-09-11T05:51:22.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149935+0000", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149998+0000", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3065999999999996e-05 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.150144+0000", 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014578400000000001 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.152779+0000", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002635025 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163265+0000", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010485758 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163327+0000", 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2415000000000005e-05 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163372+0000", 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4331e-05 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163428+0000", 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.664e-05 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.952 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6063 3.2 3:470eafa8:::rbd_data.11249b359231.0000000000000007:head [stat,truncate 3735552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.313907+0000", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.6106385650000004, 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007488181, 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6063 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313907+0000", 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313907+0000", 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313907+0000", 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9599999999999998e-07 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313911+0000", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.019e-06 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313921+0000", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4380000000000001e-06 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313927+0000", 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4880000000000004e-06 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.313983+0000", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6106000000000001e-05 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314065+0000", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1308000000000001e-05 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314201+0000", 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013640500000000001 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316654+0000", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024525720000000001 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.321337+0000", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046837149999999998 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.321369+0000", 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1665999999999998e-05 2024-09-11T05:51:22.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.321395+0000", 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5667999999999999e-05 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6095 3.b 3:d107d222:::rbd_data.11249b359231.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.5620644769999998, 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012675989, 2024-09-11T05:51:22.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6095 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362482+0000", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099e-06 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362488+0000", 2024-09-11T05:51:22.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4869999999999999e-06 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362502+0000", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3407999999999999e-05 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362510+0000", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1799999999999996e-06 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362567+0000", 2024-09-11T05:51:22.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7538999999999999e-05 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362653+0000", 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6051999999999995e-05 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364892+0000", 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022386630000000001 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.374988+0000", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010095860999999999 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375041+0000", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2785999999999999e-05 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375105+0000", 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4715999999999995e-05 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375157+0000", 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1198e-05 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6832 3.2 3:470eafa8:::rbd_data.11249b359231.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.536840+0000", 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3877049899999996, 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072106039999999998, 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.964 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6832 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536840+0000", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536840+0000", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536841+0000", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201e-06 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536848+0000", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0469999999999996e-06 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536863+0000", 2024-09-11T05:51:22.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4973e-05 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.536871+0000", 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8099999999999998e-06 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.537029+0000", 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015759899999999999 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.537194+0000", 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016493200000000001 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.538601+0000", 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014068500000000001 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.543868+0000", 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005267229 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.543896+0000", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8520000000000001e-05 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.543980+0000", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4105999999999997e-05 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.544051+0000", 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0337000000000001e-05 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.969 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6848 3.19 e33/23 3:9ed2d6d7:::rbd_object_map.11249b359231:head v 33'248, mlcod=33'246)", 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.574712+0000", 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3498334730000003, 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055789380000000003, 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574712+0000", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574712+0000", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574715+0000", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9059999999999998e-06 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574733+0000", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.889e-05 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574817+0000", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3368999999999994e-05 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.574969+0000", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015223299999999999 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.575081+0000", 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111795 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.575110+0000", 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9377999999999999e-05 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580218+0000", 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051078160000000003 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.580291+0000", 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2551000000000004e-05 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6853 3.b 3:d3b81824:::rbd_data.11249b359231.0000000000000020:head [write 2033077~14923] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.589398+0000", 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3351466470000002, 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066869119999999997, 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.974 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6853 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589398+0000", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589398+0000", 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589402+0000", 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8330000000000004e-06 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589438+0000", 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5862999999999999e-05 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589485+0000", 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7243999999999997e-05 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589516+0000", 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0642999999999998e-05 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589681+0000", 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016448100000000001 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.589693+0000", 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2544000000000001e-05 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.592724+0000", 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030305900000000001 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.592849+0000", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012545199999999999 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.593128+0000", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000279344 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.595595+0000", 2024-09-11T05:51:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024667109999999999 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.595987+0000", 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000391457 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.596040+0000", 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3850000000000001e-05 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.596085+0000", 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.49e-05 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12225 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.015587+0000", 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.90895825100000005, 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005751333, 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12225 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015587+0000", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015587+0000", 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015587+0000", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.37e-07 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015592+0000", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2679999999999996e-06 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015598+0000", 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7630000000000003e-06 2024-09-11T05:51:22.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015603+0000", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8670000000000002e-06 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.016230+0000", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062695700000000004 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.016332+0000", 2024-09-11T05:51:22.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101819 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.016693+0000", 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036139700000000001 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.017573+0000", 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00088003699999999996 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021240+0000", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036669900000000002 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021309+0000", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8532e-05 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021338+0000", 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9166000000000002e-05 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12226 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.015612+0000", 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.90893263199999996, 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058601840000000001, 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12226 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015612+0000", 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015612+0000", 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015613+0000", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.46e-07 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015615+0000", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.182e-06 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015621+0000", 2024-09-11T05:51:22.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8619999999999999e-06 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.015625+0000", 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6679999999999999e-06 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.017014+0000", 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013891439999999999 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.017070+0000", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6576000000000002e-05 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.017277+0000", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000206498 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.018202+0000", 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00092490300000000003 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021388+0000", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031866540000000001 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021434+0000", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5961999999999998e-05 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.021473+0000", 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8288999999999999e-05 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.992 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12375 3.b 3:d592f06e:::rbd_data.1124518b2c12.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 3542259~8433] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.83909034100000002, 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010649456999999999, 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12375 2024-09-11T05:51:22.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085456+0000", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999991e-07 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086709+0000", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012536859999999999 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086732+0000", 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2459999999999998e-05 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086742+0000", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6330000000000008e-06 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086807+0000", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5780999999999999e-05 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086935+0000", 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012802700000000001 2024-09-11T05:51:22.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.087312+0000", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000376948 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.095385+0000", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080731789999999998 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096024+0000", 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063872600000000005 2024-09-11T05:51:22.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096068+0000", 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3427999999999998e-05 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096104+0000", 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6628999999999998e-05 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:22.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12436 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.71478114199999998, 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095948849999999992, 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12436 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:22.999 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209766+0000", 2024-09-11T05:51:23.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5170000000000002e-06 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209796+0000", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9689999999999999e-05 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209831+0000", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4913000000000003e-05 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209855+0000", 2024-09-11T05:51:23.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3643e-05 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.210745+0000", 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089045300000000003 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.210963+0000", 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218068 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.213370+0000", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002406975 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.215267+0000", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018967910000000001 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219268+0000", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040012959999999997 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219307+0000", 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9042999999999999e-05 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219359+0000", 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1496000000000003e-05 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12439 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:23.004 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.71463734300000004, 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011043397999999999, 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12439 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:23.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209909+0000", 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6729999999999999e-06 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209919+0000", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1570000000000006e-06 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209949+0000", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0029999999999999e-05 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209965+0000", 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6290000000000002e-05 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214405+0000", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044401609999999998 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214509+0000", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010385700000000001 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214956+0000", 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044748200000000002 2024-09-11T05:51:23.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.215653+0000", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069695599999999999 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220661+0000", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050078839999999998 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220831+0000", 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017013600000000001 2024-09-11T05:51:23.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220951+0000", 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119772 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12759 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864412+0000", 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.060132985, 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056369549999999999, 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12759 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864412+0000", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864412+0000", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864413+0000", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0860000000000001e-06 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864418+0000", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2089999999999996e-06 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864433+0000", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4239e-05 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864441+0000", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3690000000000007e-06 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864477+0000", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5513999999999999e-05 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864526+0000", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9626999999999998e-05 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864711+0000", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018444800000000001 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.868186+0000", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034754870000000002 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869941+0000", 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017551489999999999 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.870000+0000", 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8412999999999998e-05 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.870049+0000", 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9413999999999998e-05 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12760 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864457+0000", 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.060088071, 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0087030340000000001, 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12760 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864457+0000", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864457+0000", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864458+0000", 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3499999999999996e-07 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864461+0000", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5779999999999999e-06 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864470+0000", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3049999999999993e-06 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864475+0000", 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690000000000001e-06 2024-09-11T05:51:23.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.865049+0000", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057319800000000002 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.865140+0000", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1686000000000003e-05 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.865512+0000", 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037171099999999999 2024-09-11T05:51:23.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869068+0000", 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035556170000000001 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873056+0000", 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039881730000000002 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873117+0000", 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1014999999999998e-05 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873160+0000", 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3247000000000001e-05 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12761 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:23.021 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864487+0000", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.060058421000000001, 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088373159999999996, 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12761 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864487+0000", 2024-09-11T05:51:23.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864487+0000", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864487+0000", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.61e-07 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864491+0000", 2024-09-11T05:51:23.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4570000000000001e-06 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864501+0000", 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005e-05 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864506+0000", 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.866628+0000", 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021217419999999998 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.866718+0000", 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0043000000000005e-05 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.866941+0000", 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000223031 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869552+0000", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026108799999999999 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873218+0000", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036656039999999998 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873278+0000", 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9991000000000002e-05 2024-09-11T05:51:23.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873324+0000", 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6313999999999997e-05 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12762 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.060029908999999999, 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094195340000000002, 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12762 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864516+0000", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864524+0000", 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2330000000000006e-06 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864532+0000", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0059999999999992e-06 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864537+0000", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5300000000000004e-06 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867222+0000", 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002684475 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867266+0000", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4604999999999997e-05 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867445+0000", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017825499999999999 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869566+0000", 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021212119999999999 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873837+0000", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042710760000000004 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873892+0000", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4981000000000001e-05 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873935+0000", 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2710999999999998e-05 2024-09-11T05:51:23.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12763 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.059999522, 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098476699999999993, 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:23.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12763 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:23.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0500000000000004e-07 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864549+0000", 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8389999999999998e-06 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:23.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864556+0000", 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1949999999999997e-06 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864561+0000", 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3240000000000002e-06 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867714+0000", 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031524370000000001 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867758+0000", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3903000000000003e-05 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867934+0000", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017616499999999999 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869572+0000", 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016385309999999999 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874165+0000", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045921989999999999 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874236+0000", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1491999999999999e-05 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874393+0000", 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015708000000000001 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:23.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:23.039 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:23.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:23.040 INFO:teuthology.orchestra.run.smithi165.stdout:1377 write 0xc5761b7 thru 0xc58175c (0xb5a6 bytes) 2024-09-11T05:51:23.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1378 writesame 0xe8b6209 thru 0xe8bfd4b data_size 0x1(0x9b43 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1382 write 0xda4a7a6 thru 0xda4bbb5 (0x1410 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1383 write 0x296dab thru 0x2a6450 (0xf6a6 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1384 write 0x7b5a463 thru 0x7b663e9 (0xbf87 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1385 punch from 0x978117b to 0x978532e, (0x41b3 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1386 writesame 0x5c4a3e6 thru 0x5c54959 data_size 0x2(0xa574 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1387 read 0xb192a4b thru 0xb1a279c (0xfd52 bytes) 2024-09-11T05:51:23.042 INFO:teuthology.orchestra.run.smithi165.stdout:1388 trunc from 0xe8bfd4c to 0x6d55000 2024-09-11T05:51:23.043 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:23.042+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:23.062 INFO:teuthology.orchestra.run.smithi165.stdout:1389 write 0xa777c9f thru 0xa77ad97 (0x30f9 bytes) 2024-09-11T05:51:23.071 INFO:teuthology.orchestra.run.smithi165.stdout:1390 writesame 0x5804414 thru 0x5808271 data_size 0x9(0x3e5e bytes) 2024-09-11T05:51:23.089 INFO:teuthology.orchestra.run.smithi165.stdout:1391 write 0xc27bfc2 thru 0xc27f87b (0x38ba bytes) 2024-09-11T05:51:23.096 INFO:teuthology.orchestra.run.smithi165.stdout:1392 trunc from 0xc27f87c to 0xd9ed31d 2024-09-11T05:51:23.102 INFO:teuthology.orchestra.run.smithi165.stdout:1393 clone 7 order 25 su 131072 sc 15 2024-09-11T05:51:23.141 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:23.142+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:23.174 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:23.175 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:23.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:23.218 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:23.219 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:23.235 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:23.235 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:23.250 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:23.251 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:23.266 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:23.266 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:23.276 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:23.278+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:23.283 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:23.284 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:23.299 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:23.300 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:23.318 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:23.318 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:23.338 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:23.338 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:23.342 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:23.338+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:23.355 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:23.355 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:23.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:23.444 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:23.442+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:23.542 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:23.542 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:23.544 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:23.546+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:23.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:23.613 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:51:23.622 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T05:51:23.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T05:51:23.645 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:23.646+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:23.730 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:23.730 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:23.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:23.746 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:23.746+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:23.755 INFO:teuthology.orchestra.run.smithi165.stdout:flattening image image_client.0-clone6 2024-09-11T05:51:23.847 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:23.846+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:23.877 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:23.878 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:23.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:23.980 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:23.982+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:24.018 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-09-11T05:51:24.049 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:24.050+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:24.062 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.062 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:24.118 INFO:teuthology.orchestra.run.smithi165.stdout:1394 trunc from 0xd9ed31d to 0xb019aab 2024-09-11T05:51:24.169 INFO:teuthology.orchestra.run.smithi165.stdout:1395 punch from 0xae65911 to 0xae6f477, (0x9b66 bytes) 2024-09-11T05:51:24.171 INFO:teuthology.orchestra.run.smithi165.stdout:1397 trunc from 0xb019aab to 0xbec9724 2024-09-11T05:51:24.172 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:24.174+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:24.182 INFO:teuthology.orchestra.run.smithi165.stdout:1398 read 0x3bc0291 thru 0x3bc16d2 (0x1442 bytes) 2024-09-11T05:51:24.183 INFO:teuthology.orchestra.run.smithi165.stdout:1399 read 0x7a34ad6 thru 0x7a40966 (0xbe91 bytes) 2024-09-11T05:51:24.184 INFO:teuthology.orchestra.run.smithi165.stdout:1401 punch from 0x354d567 to 0x3557001, (0x9a9a bytes) 2024-09-11T05:51:24.189 INFO:teuthology.orchestra.run.smithi165.stdout:1402 trunc from 0xbec9724 to 0xe3e1173 2024-09-11T05:51:24.201 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.201 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.201 INFO:teuthology.orchestra.run.smithi165.stdout:1406 write 0x9ffcbb thru 0xa0dac4 (0xde0a bytes) 2024-09-11T05:51:24.204 INFO:teuthology.orchestra.run.smithi165.stdout:1407 writesame 0x9ed4668 thru 0x9eda8f4 data_size 0x1(0x628d bytes) 2024-09-11T05:51:24.232 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.232 INFO:teuthology.orchestra.run.smithi165.stdout:1409 writesame 0xd2d027 thru 0xd32646 data_size 0x35(0x5620 bytes) 2024-09-11T05:51:24.236 INFO:teuthology.orchestra.run.smithi165.stdout:1410 read 0xaa0d103 thru 0xaa198d0 (0xc7ce bytes) 2024-09-11T05:51:24.237 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.237 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.237 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.237 INFO:teuthology.orchestra.run.smithi165.stdout:1414 read 0xae9d914 thru 0xae9dc05 (0x2f2 bytes) 2024-09-11T05:51:24.238 INFO:teuthology.orchestra.run.smithi165.stdout:1415 write 0x2d6ebf1 thru 0x2d71125 (0x2535 bytes) 2024-09-11T05:51:24.241 INFO:teuthology.orchestra.run.smithi165.stdout:1416 write 0x8bab8b3 thru 0x8bb404e (0x879c bytes) 2024-09-11T05:51:24.244 INFO:teuthology.orchestra.run.smithi165.stdout:1418 write 0x64625ed thru 0x646e5d4 (0xbfe8 bytes) 2024-09-11T05:51:24.248 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.248 INFO:teuthology.orchestra.run.smithi165.stdout:1421 write 0x628183f thru 0x62915d2 (0xfd94 bytes) 2024-09-11T05:51:24.250 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:24.246+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:24.251 INFO:teuthology.orchestra.run.smithi165.stdout:1422 write 0xba7cb15 thru 0xba87d26 (0xb212 bytes) 2024-09-11T05:51:24.255 INFO:teuthology.orchestra.run.smithi165.stdout:1423 writesame 0xace3517 thru 0xacedd86 data_size 0x2c(0xa870 bytes) 2024-09-11T05:51:24.256 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.256 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.258 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:24.259 INFO:teuthology.orchestra.run.smithi165.stdout:1425 read 0xe1caec1 thru 0xe1d5e99 (0xafd9 bytes) 2024-09-11T05:51:24.259 INFO:teuthology.orchestra.run.smithi165.stdout:1426 read 0xd55bf96 thru 0xd5638fa (0x7965 bytes) 2024-09-11T05:51:24.260 INFO:teuthology.orchestra.run.smithi165.stdout:1427 writesame 0x68defab thru 0x68e7d43 data_size 0x2b(0x8d99 bytes) 2024-09-11T05:51:24.265 INFO:teuthology.orchestra.run.smithi165.stdout:1428 writesame 0x2a19234 thru 0x2a20f49 data_size 0x12(0x7d16 bytes) 2024-09-11T05:51:24.270 INFO:teuthology.orchestra.run.smithi165.stdout:1429 write 0x3dad53f thru 0x3dbb54c (0xe00e bytes) 2024-09-11T05:51:24.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:24.273 INFO:teuthology.orchestra.run.smithi165.stdout:1431 writesame 0x37416c thru 0x37ccba data_size 0x1(0x8b4f bytes) 2024-09-11T05:51:24.302 INFO:teuthology.orchestra.run.smithi165.stdout:1433 read 0x5eafce7 thru 0x5eafdad (0xc7 bytes) 2024-09-11T05:51:24.303 INFO:teuthology.orchestra.run.smithi165.stdout:1434 writesame 0x401aa0 thru 0x401e6b data_size 0x9(0x3cc bytes) 2024-09-11T05:51:24.306 INFO:teuthology.orchestra.run.smithi165.stdout:1435 writesame 0x2b190b4 thru 0x2b23a67 data_size 0x2(0xa9b4 bytes) 2024-09-11T05:51:24.318 INFO:teuthology.orchestra.run.smithi165.stdout:1436 write 0x79b545b thru 0x79b5514 (0xba bytes) 2024-09-11T05:51:24.321 INFO:teuthology.orchestra.run.smithi165.stdout:1437 write 0x1e888ed thru 0x1e9511a (0xc82e bytes) 2024-09-11T05:51:24.325 INFO:teuthology.orchestra.run.smithi165.stdout:1438 write 0x7d94d65 thru 0x7da3d8d (0xf029 bytes) 2024-09-11T05:51:24.330 INFO:teuthology.orchestra.run.smithi165.stdout:1440 punch from 0x8a61573 to 0x8a6c585, (0xb012 bytes) 2024-09-11T05:51:24.330 INFO:teuthology.orchestra.run.smithi165.stdout:1441 read 0x7d08a3 thru 0x7d4c6c (0x43ca bytes) 2024-09-11T05:51:24.331 INFO:teuthology.orchestra.run.smithi165.stdout:1442 write 0xbf9d437 thru 0xbfaa242 (0xce0c bytes) 2024-09-11T05:51:24.334 INFO:teuthology.orchestra.run.smithi165.stdout:1443 clone 8 order 22 su 524288 sc 11 2024-09-11T05:51:24.351 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:24.350+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:24.432 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.432 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:24.452 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:24.450+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:24.553 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:24.554+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:24.631 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T05:51:24.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T05:51:24.672 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:24.674+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:24.724 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone7 from 0xe3e1173 (overlap 0xb019aab) to 0x0 2024-09-11T05:51:24.755 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:24.754+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:24.756 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.756 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:24.802 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-09-11T05:51:24.857 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:24.854+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:24.938 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:24.938 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:24.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:24.958 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:24.954+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:25.059 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:25.058+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:25.074 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:25.074 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:25.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:25.159 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:25.162+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:25.199 INFO:teuthology.orchestra.run.smithi165.stdout:1445 write 0x1741014 thru 0x174b7a8 (0xa795 bytes) 2024-09-11T05:51:25.214 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:25.214 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:25.216 INFO:teuthology.orchestra.run.smithi165.stdout:1446 writesame 0xb6038fa thru 0xb60dc4e data_size 0x1(0xa355 bytes) 2024-09-11T05:51:25.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:25.276 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:25.274+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:25.278 INFO:teuthology.orchestra.run.smithi165.stdout:1447 writesame 0x1e3212c thru 0x1e3e29b data_size 0x28(0xc170 bytes) 2024-09-11T05:51:25.287 INFO:teuthology.orchestra.run.smithi165.stdout:1448 writesame 0x9ea49fc thru 0x9eaf963 data_size 0xc(0xaf68 bytes) 2024-09-11T05:51:25.312 INFO:teuthology.orchestra.run.smithi165.stdout:1449 trunc from 0xe3e1173 to 0x6a33e40 2024-09-11T05:51:25.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:25.366 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:25.366 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:25.366 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:25.366+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:25.372 INFO:teuthology.orchestra.run.smithi165.stdout:1450 write 0x58c720c thru 0x58d3655 (0xc44a bytes) 2024-09-11T05:51:25.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:25.386 INFO:teuthology.orchestra.run.smithi165.stdout:1451 write 0x6f29344 thru 0x6f33e1e (0xaadb bytes) 2024-09-11T05:51:25.388 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.388 INFO:teuthology.orchestra.run.smithi165.stdout:1455 writesame 0x20a2907 thru 0x20a69dc data_size 0x2b(0x40d6 bytes) 2024-09-11T05:51:25.397 INFO:teuthology.orchestra.run.smithi165.stdout:1457 writesame 0x2769751 thru 0x2772904 data_size 0xa(0x91b4 bytes) 2024-09-11T05:51:25.409 INFO:teuthology.orchestra.run.smithi165.stdout:1461 read 0x34a2f0e thru 0x34ad249 (0xa33c bytes) 2024-09-11T05:51:25.429 INFO:teuthology.orchestra.run.smithi165.stdout:1463 writesame 0xcf12b93 thru 0xcf133ce data_size 0x11(0x83c bytes) 2024-09-11T05:51:25.429 INFO:teuthology.orchestra.run.smithi165.stdout:1464 write 0xbd4ac26 thru 0xbd5a228 (0xf603 bytes) 2024-09-11T05:51:25.435 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.435 INFO:teuthology.orchestra.run.smithi165.stdout:1466 punch from 0x526c635 to 0x52723fb, (0x5dc6 bytes) 2024-09-11T05:51:25.436 INFO:teuthology.orchestra.run.smithi165.stdout:1467 read 0x24d2d2d thru 0x24db161 (0x8435 bytes) 2024-09-11T05:51:25.436 INFO:teuthology.orchestra.run.smithi165.stdout:1469 write 0x17a346f thru 0x17aaa78 (0x760a bytes) 2024-09-11T05:51:25.448 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.448 INFO:teuthology.orchestra.run.smithi165.stdout:1471 write 0x2ac4dd1 thru 0x2acdb3c (0x8d6c bytes) 2024-09-11T05:51:25.459 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.459 INFO:teuthology.orchestra.run.smithi165.stdout:1473 punch from 0x4d7890f to 0x4d82d9c, (0xa48d bytes) 2024-09-11T05:51:25.459 INFO:teuthology.orchestra.run.smithi165.stdout:1475 read 0x82785a8 thru 0x8284de4 (0xc83d bytes) 2024-09-11T05:51:25.459 INFO:teuthology.orchestra.run.smithi165.stdout:1476 writesame 0x2d93cc4 thru 0x2d9b009 data_size 0xd(0x7346 bytes) 2024-09-11T05:51:25.462 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:25.458+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:25.474 INFO:teuthology.orchestra.run.smithi165.stdout:1477 punch from 0x57bcee0 to 0x57bff91, (0x30b1 bytes) 2024-09-11T05:51:25.474 INFO:teuthology.orchestra.run.smithi165.stdout:1478 punch from 0x77aa15b to 0x77aeb09, (0x49ae bytes) 2024-09-11T05:51:25.474 INFO:teuthology.orchestra.run.smithi165.stdout:1487 write 0x66a7cd4 thru 0x66b6046 (0xe373 bytes) 2024-09-11T05:51:25.476 INFO:teuthology.orchestra.run.smithi165.stdout:1488 read 0x62c469c thru 0x62cd6d9 (0x903e bytes) 2024-09-11T05:51:25.477 INFO:teuthology.orchestra.run.smithi165.stdout:1490 read 0x41d5a7c thru 0x41e336d (0xd8f2 bytes) 2024-09-11T05:51:25.478 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.479 INFO:teuthology.orchestra.run.smithi165.stdout:1493 trunc from 0xcf133cf to 0xdada82a 2024-09-11T05:51:25.486 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.487 INFO:teuthology.orchestra.run.smithi165.stdout:1496 write 0x9cfb033 thru 0x9d0a7bd (0xf78b bytes) 2024-09-11T05:51:25.494 INFO:teuthology.orchestra.run.smithi165.stdout:1497 write 0x5bd5889 thru 0x5bdd5ab (0x7d23 bytes) 2024-09-11T05:51:25.496 INFO:teuthology.orchestra.run.smithi165.stdout:1498 write 0x73547ee thru 0x7361c43 (0xd456 bytes) 2024-09-11T05:51:25.501 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.501 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.507 INFO:teuthology.orchestra.run.smithi165.stdout:1503 trunc from 0xdada82a to 0x7885522 2024-09-11T05:51:25.509 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:25.509 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:25.519 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1064 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598650862, 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01221121, 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1064 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921456+0000", 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921457+0000", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.108e-06 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921463+0000", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.981e-06 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921476+0000", 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3149999999999999e-05 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921484+0000", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0739999999999993e-06 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922559+0000", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010747560000000001 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.922816+0000", 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025728099999999999 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.923880+0000", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010636319999999999 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929525+0000", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056459359999999998 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933368+0000", 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038422479999999999 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933523+0000", 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015488899999999999 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933667+0000", 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144155 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.525 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1065 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598608394999999, 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012476767, 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1065 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921498+0000", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921499+0000", 2024-09-11T05:51:25.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1099999999999995e-07 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921506+0000", 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8259999999999993e-06 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921515+0000", 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921522+0000", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.924898+0000", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033757380000000001 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925011+0000", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113663 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.925372+0000", 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036081200000000001 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929538+0000", 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041663910000000002 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933820+0000", 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042815140000000002 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933911+0000", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0989000000000006e-05 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933975+0000", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3806000000000005e-05 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1066 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:25.531 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598514222, 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012635749, 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1066 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921592+0000", 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921593+0000", 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5099999999999998e-07 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921599+0000", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4609999999999999e-06 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921609+0000", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0434e-05 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921617+0000", 2024-09-11T05:51:25.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2970000000000004e-06 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926267+0000", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046498060000000003 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926348+0000", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0680999999999995e-05 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.926715+0000", 2024-09-11T05:51:25.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000367525 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932135+0000", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054191999999999999 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934076+0000", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019415319999999999 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934172+0000", 2024-09-11T05:51:25.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5725999999999998e-05 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934228+0000", 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6135999999999999e-05 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1067 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598387002999999, 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012883237000000001, 2024-09-11T05:51:25.537 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1067 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921719+0000", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921720+0000", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4799999999999997e-07 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921725+0000", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7450000000000003e-06 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921734+0000", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3079999999999997e-06 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921743+0000", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5609999999999999e-06 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927241+0000", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054980680000000001 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.927487+0000", 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000245758 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.928564+0000", 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010776990000000001 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.933718+0000", 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051533330000000004 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934356+0000", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063792199999999997 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934504+0000", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014797200000000001 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934603+0000", 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9023000000000004e-05 2024-09-11T05:51:25.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1068 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598304604999999, 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015284591, 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1068 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921802+0000", 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0500000000000003e-07 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921807+0000", 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9919999999999998e-06 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921816+0000", 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0999999999999993e-06 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921824+0000", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.255e-06 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.929887+0000", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080631929999999997 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.930128+0000", 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241318 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.931196+0000", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001067709 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934254+0000", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030585059999999999 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.936944+0000", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002689126 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937038+0000", 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4772999999999995e-05 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.937086+0000", 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8013999999999998e-05 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1069 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.548 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.598239834999999, 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017393289999999999, 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1069 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921867+0000", 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:51:25.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921872+0000", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6480000000000002e-06 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921880+0000", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9850000000000006e-06 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.921887+0000", 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932217+0000", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010329680000000001 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932347+0000", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129701 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.932757+0000", 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000409904 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.934275+0000", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015181579999999999 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939164+0000", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048894680000000001 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939216+0000", 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1721999999999997e-05 2024-09-11T05:51:25.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:12.939260+0000", 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3782000000000002e-05 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1132 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.485991859, 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011988883000000001, 2024-09-11T05:51:25.554 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1132 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034115+0000", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-07 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034119+0000", 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1919999999999998e-06 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034132+0000", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3036e-05 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034145+0000", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2849e-05 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038262+0000", 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041172459999999998 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038341+0000", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8678999999999997e-05 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.038722+0000", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038121999999999999 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043113+0000", 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004391112 2024-09-11T05:51:25.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.045962+0000", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028480300000000001 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046045+0000", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3294999999999994e-05 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046103+0000", 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8597999999999997e-05 2024-09-11T05:51:25.559 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1133 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.485946409, 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012144546000000001, 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.560 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1133 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034160+0000", 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034161+0000", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.02e-07 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034166+0000", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3090000000000003e-06 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034174+0000", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.034180+0000", 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0100000000000001e-06 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039023+0000", 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048429850000000002 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039070+0000", 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7558000000000001e-05 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.039252+0000", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000182051 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.043138+0000", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003885234 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046192+0000", 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030545910000000002 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046257+0000", 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4529000000000005e-05 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.046305+0000", 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7846000000000002e-05 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1351 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:25.565 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.87058362, 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012039169000000001, 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1351 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649523+0000", 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4200000000000001e-07 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649526+0000", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2389999999999998e-06 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649535+0000", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2279999999999998e-06 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649540+0000", 2024-09-11T05:51:25.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3759999999999999e-06 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657820+0000", 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082799009999999992 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.657923+0000", 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103447 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658388+0000", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046494 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659901+0000", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015124719999999999 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661457+0000", 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001555842 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661517+0000", 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9917999999999998e-05 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.661562+0000", 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5364000000000001e-05 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1352 3.1f 3:f8019b30:::rbd_object_map.112475afb12f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-11T05:51:25.571 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.870559522000001, 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014094584, 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1352 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:25.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649547+0000", 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7099999999999997e-07 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649550+0000", 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6280000000000001e-06 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649556+0000", 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.246e-06 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.649561+0000", 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6630000000000001e-06 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.658923+0000", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093620470000000001 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659009+0000", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5964000000000006e-05 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.659436+0000", 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042755199999999999 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.660650+0000", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001213385 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663539+0000", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002889165 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663595+0000", 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6165999999999997e-05 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.663641+0000", 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6397000000000003e-05 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10127 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7349479749999999, 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011864169000000001, 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10127 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785158+0000", 2024-09-11T05:51:25.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785159+0000", 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0400000000000002e-07 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785161+0000", 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9400000000000001e-06 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785167+0000", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0179999999999998e-06 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785171+0000", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4649999999999996e-06 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796850+0000", 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011679172 2024-09-11T05:51:25.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.796923+0000", 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2836000000000005e-05 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797023+0000", 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9333999999999995e-05 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10128 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3866624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7349291249999998, 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012011195000000001, 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10128 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785177+0000", 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785178+0000", 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7899999999999999e-07 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785185+0000", 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1570000000000003e-06 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785191+0000", 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3489999999999997e-06 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785196+0000", 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8189999999999999e-06 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797053+0000", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011856716999999999 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797126+0000", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3351999999999998e-05 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797188+0000", 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2422000000000006e-05 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10129 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.734903847, 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012703261, 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10129 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.586 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785203+0000", 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8000000000000001e-07 2024-09-11T05:51:25.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785205+0000", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9769999999999999e-06 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785211+0000", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9630000000000003e-06 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785216+0000", 2024-09-11T05:51:25.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.656e-06 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797215+0000", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011999697 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797819+0000", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060352099999999998 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797906+0000", 2024-09-11T05:51:25.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7066999999999998e-05 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10130 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7348846660000001, 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012870727, 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10130 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785222+0000", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785224+0000", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.945e-06 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785231+0000", 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4670000000000001e-06 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785237+0000", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1759999999999998e-06 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.797938+0000", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012701097999999999 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798016+0000", 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7873000000000005e-05 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798092+0000", 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6749000000000007e-05 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10131 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7348636659999999, 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013020771, 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10131 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785243+0000", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0200000000000003e-07 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785245+0000", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.838e-06 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785251+0000", 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2090000000000002e-06 2024-09-11T05:51:25.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785256+0000", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5029999999999999e-06 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798122+0000", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012866803 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798199+0000", 2024-09-11T05:51:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6730000000000006e-05 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798263+0000", 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4286000000000001e-05 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:10132 3.1b 3:dd0a3ff4:::rbd_data.11249b359231.0000000000000009:7 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.73482687, 2024-09-11T05:51:25.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013218944, 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 10132 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785280+0000", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9900000000000001e-07 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785283+0000", 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8389999999999998e-06 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785297+0000", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4245e-05 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.785302+0000", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3970000000000002e-06 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798345+0000", 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013042321000000001 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798421+0000", 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6666000000000001e-05 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:20.798498+0000", 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6977000000000005e-05 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11483 3.17 3:ef8497a9:::rbd_data.1124518b2c12.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 3396905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4096382829999996, 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014624382999999999, 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11483 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110468+0000", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.110471+0000", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111721+0000", 2024-09-11T05:51:25.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001250323 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111747+0000", 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6353999999999999e-05 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111760+0000", 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2962000000000001e-05 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111830+0000", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0217999999999998e-05 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.111968+0000", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013758800000000001 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.112530+0000", 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056166300000000001 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.121334+0000", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088046630000000008 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.124955+0000", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036208249999999998 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.125043+0000", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8189000000000006e-05 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.125092+0000", 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9191999999999999e-05 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12580 3.1 3:8726497b:::rbd_data.1124518b2c12.0000000000000025:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.374211+0000", 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.145895216, 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030165040000000001, 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12580 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374211+0000", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374211+0000", 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374214+0000", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.723e-06 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374229+0000", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4927e-05 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374262+0000", 2024-09-11T05:51:25.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3340000000000003e-05 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374278+0000", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374398+0000", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120036 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.374535+0000", 2024-09-11T05:51:25.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136222 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.380166+0000", 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056309139999999999 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404251+0000", 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024085691999999999 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404267+0000", 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5509999999999999e-05 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404313+0000", 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6205000000000002e-05 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404376+0000", 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3225000000000005e-05 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:12580 3.1 e37/23 ondisk, result = 0)", 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.389378+0000", 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1307286599999999, 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014881168, 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389378+0000", 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389378+0000", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389380+0000", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.804e-06 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389385+0000", 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0340000000000004e-06 2024-09-11T05:51:25.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389402+0000", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7461000000000001e-05 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.389411+0000", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4609999999999997e-06 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404236+0000", 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014824739999999999 2024-09-11T05:51:25.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404246+0000", 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3680000000000008e-06 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.404259+0000", 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-05 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12694 3.17 3:ef8497a9:::rbd_data.1124518b2c12.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.674907+0000", 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.845199467, 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022492806000000001, 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12694 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674907+0000", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674907+0000", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674909+0000", 2024-09-11T05:51:25.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4499999999999998e-06 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674923+0000", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3801000000000001e-05 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674950+0000", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6497000000000001e-05 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.674963+0000", 2024-09-11T05:51:25.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3528e-05 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.675073+0000", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109885 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.675195+0000", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122082 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.679534+0000", 2024-09-11T05:51:25.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043388239999999998 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.697274+0000", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017739522000000001 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.697313+0000", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9199000000000002e-05 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.697360+0000", 2024-09-11T05:51:25.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7063e-05 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.697400+0000", 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9954999999999999e-05 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:25.623 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:25.624 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:25.624 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:25.624 INFO:teuthology.orchestra.run.smithi165.stdout:1504 trunc from 0x7885522 to 0x7062df9 2024-09-11T05:51:25.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:25.626 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:25.626 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:25.626 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stdout:1508 trunc from 0x7062df9 to 0xbd8f4dd 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stdout:1509 write 0x22bbd58 thru 0x22cbabc (0xfd65 bytes) 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:25.627 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:25.628 INFO:teuthology.orchestra.run.smithi165.stdout:1510 read 0x7fbe28c thru 0x7fcc736 (0xe4ab bytes) 2024-09-11T05:51:25.628 INFO:teuthology.orchestra.run.smithi165.stdout:1511 write 0x1d79221 thru 0x1d88877 (0xf657 bytes) 2024-09-11T05:51:25.628 INFO:teuthology.orchestra.run.smithi165.stdout:1512 write 0x38dbbd3 thru 0x38dc246 (0x674 bytes) 2024-09-11T05:51:25.628 INFO:teuthology.orchestra.run.smithi165.stdout:1513 punch from 0x56cedcb to 0x56d7140, (0x8375 bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:1515 writesame 0x5dc71db thru 0x5dd6208 data_size 0x47(0xf02e bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:1517 read 0x1302a3c thru 0x13070d7 (0x469c bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:1521 punch from 0x14f8bd7 to 0x14f938c, (0x7b5 bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:1522 write 0x237444e thru 0x2379b37 (0x56ea bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:1525 punch from 0x8994b43 to 0x89a4470, (0xf92d bytes) 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:25.629 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:25.630 INFO:teuthology.orchestra.run.smithi165.stdout:1526 writesame 0xc9c9eb8 thru 0xc9cc8e7 data_size 0x24(0x2a30 bytes) 2024-09-11T05:51:25.630 INFO:teuthology.orchestra.run.smithi165.stdout:1527 write 0x8266f13 thru 0x8268fc4 (0x20b2 bytes) 2024-09-11T05:51:25.630 INFO:teuthology.orchestra.run.smithi165.stdout:1528 read 0x6ba8ffe thru 0x6bb381e (0xa821 bytes) 2024-09-11T05:51:25.630 INFO:teuthology.orchestra.run.smithi165.stdout:1530 trunc from 0xc9cc8e8 to 0x4174aff 2024-09-11T05:51:25.630 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:25.631 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:25.631 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:25.622+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:25.639 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T05:51:25.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:51:25.648 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:25.648 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:25.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T05:51:25.678 INFO:teuthology.orchestra.run.smithi165.stdout:1531 trunc from 0x4174aff to 0x8ad0166 2024-09-11T05:51:25.678 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:25.679 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:25.688 INFO:teuthology.orchestra.run.smithi165.stdout:1533 read 0x83daf7d thru 0x83e9bb2 (0xec36 bytes) 2024-09-11T05:51:25.704 INFO:teuthology.orchestra.run.smithi165.stdout:1534 writesame 0xe6d0428 thru 0xe6d8cf9 data_size 0x53(0x88d2 bytes) 2024-09-11T05:51:25.704 INFO:teuthology.orchestra.run.smithi165.stdout:1535 read 0x1008470 thru 0x1014388 (0xbf19 bytes) 2024-09-11T05:51:25.705 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.705 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.705 INFO:teuthology.orchestra.run.smithi165.stdout:1539 write 0xb6dae65 thru 0xb6e4241 (0x93dd bytes) 2024-09-11T05:51:25.712 INFO:teuthology.orchestra.run.smithi165.stdout:1540 writesame 0xb21da27 thru 0xb22323e data_size 0x4(0x5818 bytes) 2024-09-11T05:51:25.723 INFO:teuthology.orchestra.run.smithi165.stdout:1542 writesame 0x4172657 thru 0x41748d9 data_size 0x13(0x2283 bytes) 2024-09-11T05:51:25.725 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:25.722+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:25.727 INFO:teuthology.orchestra.run.smithi165.stdout:1545 write 0x5ed98a4 thru 0x5edb295 (0x19f2 bytes) 2024-09-11T05:51:25.732 INFO:teuthology.orchestra.run.smithi165.stdout:1546 writesame 0xdd38043 thru 0xdd3cadd data_size 0x1(0x4a9b bytes) 2024-09-11T05:51:25.744 INFO:teuthology.orchestra.run.smithi165.stdout:1548 read 0x2c6ce9a thru 0x2c6f54f (0x26b6 bytes) 2024-09-11T05:51:25.745 INFO:teuthology.orchestra.run.smithi165.stdout:1549 write 0xd67d8fd thru 0xd67ec7d (0x1381 bytes) 2024-09-11T05:51:25.764 INFO:teuthology.orchestra.run.smithi165.stdout:1550 trunc from 0xe6d8cfa to 0x5583ba6 2024-09-11T05:51:25.787 INFO:teuthology.orchestra.run.smithi165.stdout:1551 read 0x3bdf0b9 thru 0x3be8578 (0x94c0 bytes) 2024-09-11T05:51:25.795 INFO:teuthology.orchestra.run.smithi165.stdout:1552 trunc from 0x5583ba6 to 0x957a53a 2024-09-11T05:51:25.802 INFO:teuthology.orchestra.run.smithi165.stdout:1554 write 0x257cb8a thru 0x25800ad (0x3524 bytes) 2024-09-11T05:51:25.820 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:25.820 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:25.820 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:25.820 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:25.824 INFO:teuthology.orchestra.run.smithi165.stdout:1556 trunc from 0x957a53a to 0x43f6278 2024-09-11T05:51:25.826 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:25.822+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:25.838 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.839 INFO:teuthology.orchestra.run.smithi165.stdout:1558 write 0x26a9511 thru 0x26b1631 (0x8121 bytes) 2024-09-11T05:51:25.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:25.853 INFO:teuthology.orchestra.run.smithi165.stdout:1559 read 0x3778279 thru 0x3787548 (0xf2d0 bytes) 2024-09-11T05:51:25.854 INFO:teuthology.orchestra.run.smithi165.stdout:1560 read 0x1c579f0 thru 0x1c5a2e9 (0x28fa bytes) 2024-09-11T05:51:25.879 INFO:teuthology.orchestra.run.smithi165.stdout:1561 writesame 0xecd4d34 thru 0xecdcc49 data_size 0x1(0x7f16 bytes) 2024-09-11T05:51:25.879 INFO:teuthology.orchestra.run.smithi165.stdout:1562 writesame 0xf2f0e5 thru 0xf2f10f data_size 0x1(0x2b bytes) 2024-09-11T05:51:25.882 INFO:teuthology.orchestra.run.smithi165.stdout:1563 write 0x376b8a thru 0x38619f (0xf616 bytes) 2024-09-11T05:51:25.895 INFO:teuthology.orchestra.run.smithi165.stdout:1564 read 0xa351ff9 thru 0xa35bff8 (0xa000 bytes) 2024-09-11T05:51:25.895 INFO:teuthology.orchestra.run.smithi165.stdout:1566 writesame 0xdd7a779 thru 0xdd895b1 data_size 0x1(0xee39 bytes) 2024-09-11T05:51:25.927 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:25.926+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:25.929 INFO:teuthology.orchestra.run.smithi165.stdout:1567 read 0x35f5a2e thru 0x35f936e (0x3941 bytes) 2024-09-11T05:51:25.930 INFO:teuthology.orchestra.run.smithi165.stdout:1568 trunc from 0xecdcc4a to 0xeb18858 2024-09-11T05:51:25.937 INFO:teuthology.orchestra.run.smithi165.stdout:1569 punch from 0xe2acb52 to 0xe2b857c, (0xba2a bytes) 2024-09-11T05:51:25.938 INFO:teuthology.orchestra.run.smithi165.stdout:1570 writesame 0x7018f8f thru 0x701934d data_size 0x1(0x3bf bytes) 2024-09-11T05:51:25.944 INFO:teuthology.orchestra.run.smithi165.stdout:1571 write 0x5b29b1c thru 0x5b2f0a5 (0x558a bytes) 2024-09-11T05:51:25.957 INFO:teuthology.orchestra.run.smithi165.stdout:1572 write 0xeb28881 thru 0xeb2ceb7 (0x4637 bytes) 2024-09-11T05:51:25.963 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.963 INFO:teuthology.orchestra.run.smithi165.stdout:1575 read 0xcaeb7a4 thru 0xcafa710 (0xef6d bytes) 2024-09-11T05:51:25.963 INFO:teuthology.orchestra.run.smithi165.stdout:1577 read 0xbbbfe76 thru 0xbbce6fd (0xe888 bytes) 2024-09-11T05:51:25.963 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.963 INFO:teuthology.orchestra.run.smithi165.stdout:1580 write 0x19d89c thru 0x1a1fd0 (0x4735 bytes) 2024-09-11T05:51:25.966 INFO:teuthology.orchestra.run.smithi165.stdout:1581 read 0xe283507 thru 0xe28e8ac (0xb3a6 bytes) 2024-09-11T05:51:25.967 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:25.967 INFO:teuthology.orchestra.run.smithi165.stdout:1584 read 0x1505ae1 thru 0x1513619 (0xdb39 bytes) 2024-09-11T05:51:25.967 INFO:teuthology.orchestra.run.smithi165.stdout:1585 read 0xd3d106f thru 0xd3d2f27 (0x1eb9 bytes) 2024-09-11T05:51:25.981 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:25.981 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:25.981 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:25.985 INFO:teuthology.orchestra.run.smithi165.stdout:1586 trunc from 0xeb2ceb8 to 0xa84ac1 2024-09-11T05:51:25.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:26.025 INFO:teuthology.orchestra.run.smithi165.stdout:1588 write 0x34d54fa thru 0x34dad83 (0x588a bytes) 2024-09-11T05:51:26.027 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:26.026+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:26.030 INFO:teuthology.orchestra.run.smithi165.stdout:1590 punch from 0x3038251 to 0x3044ca0, (0xca4f bytes) 2024-09-11T05:51:26.053 INFO:teuthology.orchestra.run.smithi165.stdout:1591 write 0xd9d1afd thru 0xd9daec4 (0x93c8 bytes) 2024-09-11T05:51:26.058 INFO:teuthology.orchestra.run.smithi165.stdout:1593 write 0x141617e thru 0x1417be5 (0x1a68 bytes) 2024-09-11T05:51:26.061 INFO:teuthology.orchestra.run.smithi165.stdout:1596 write 0x7fe8c3e thru 0x7fe8e21 (0x1e4 bytes) 2024-09-11T05:51:26.065 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.066 INFO:teuthology.orchestra.run.smithi165.stdout:1598 trunc from 0xd9daec5 to 0xe31f4aa 2024-09-11T05:51:26.072 INFO:teuthology.orchestra.run.smithi165.stdout:1599 write 0x1580734 thru 0x1583d52 (0x361f bytes) 2024-09-11T05:51:26.075 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.075 INFO:teuthology.orchestra.run.smithi165.stdout:1604 write 0xbc3942b thru 0xbc45183 (0xbd59 bytes) 2024-09-11T05:51:26.083 INFO:teuthology.orchestra.run.smithi165.stdout:1606 read 0x7c2d8be thru 0x7c2f59e (0x1ce1 bytes) 2024-09-11T05:51:26.083 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.083 INFO:teuthology.orchestra.run.smithi165.stdout:1609 punch from 0xe95693 to 0xe9a6e2, (0x504f bytes) 2024-09-11T05:51:26.083 INFO:teuthology.orchestra.run.smithi165.stdout:1610 punch from 0x49b284b to 0x49c0cff, (0xe4b4 bytes) 2024-09-11T05:51:26.083 INFO:teuthology.orchestra.run.smithi165.stdout:1611 write 0xbce982d thru 0xbcef649 (0x5e1d bytes) 2024-09-11T05:51:26.088 INFO:teuthology.orchestra.run.smithi165.stdout:1612 read 0x445160b thru 0x4458481 (0x6e77 bytes) 2024-09-11T05:51:26.092 INFO:teuthology.orchestra.run.smithi165.stdout:1614 trunc from 0xe31f4aa to 0xaf1c44c 2024-09-11T05:51:26.111 INFO:teuthology.orchestra.run.smithi165.stdout:1615 write 0xaa986c4 thru 0xaaa3996 (0xb2d3 bytes) 2024-09-11T05:51:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:1616 read 0x5338cdb thru 0x533c839 (0x3b5f bytes) 2024-09-11T05:51:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:1617 writesame 0x7de3ea4 thru 0x7de6a01 data_size 0x7(0x2b5e bytes) 2024-09-11T05:51:26.124 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.124 INFO:teuthology.orchestra.run.smithi165.stdout:1619 punch from 0x78c2946 to 0x78d10c2, (0xe77c bytes) 2024-09-11T05:51:26.124 INFO:teuthology.orchestra.run.smithi165.stdout:1620 punch from 0x43f4e90 to 0x43f6c4a, (0x1dba bytes) 2024-09-11T05:51:26.124 INFO:teuthology.orchestra.run.smithi165.stdout:1621 read 0x7bc29e5 thru 0x7bc68dc (0x3ef8 bytes) 2024-09-11T05:51:26.129 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:26.130+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:26.133 INFO:teuthology.orchestra.run.smithi165.stdout:1622 write 0xb70e718 thru 0xb71c288 (0xdb71 bytes) 2024-09-11T05:51:26.139 INFO:teuthology.orchestra.run.smithi165.stdout:1623 write 0x82c5ed2 thru 0x82cc060 (0x618f bytes) 2024-09-11T05:51:26.144 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.144 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6094 3.4 3:25de1119:::rbd_data.11249b359231.0000000000000004:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:26.154 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.7904429000000004, 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011224494999999999, 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6094 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362469+0000", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362477+0000", 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:51:26.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362496+0000", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8598999999999999e-05 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362506+0000", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0336999999999999e-05 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362576+0000", 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0401000000000006e-05 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362671+0000", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4110999999999994e-05 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364415+0000", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017441049999999999 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373558+0000", 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091431189999999999 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373596+0000", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.803e-05 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373637+0000", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0706e-05 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373692+0000", 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5531000000000001e-05 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11133 3.0 3:061d780f:::rbd_data.1124518b2c12.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1243777760000002, 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03149018, 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11133 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028534+0000", 2024-09-11T05:51:26.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7850000000000001e-06 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028542+0000", 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8879999999999997e-06 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028559+0000", 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6603999999999999e-05 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028567+0000", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9019999999999999e-06 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028635+0000", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8046999999999994e-05 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028805+0000", 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016979000000000001 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.033986+0000", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051812589999999997 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059864+0000", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025877539000000001 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059924+0000", 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0327e-05 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059971+0000", 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7239e-05 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060023+0000", 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1799999999999999e-05 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.165 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:11133 3.0 e36/23 ondisk, result = 0)", 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1043447999999998, 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011468028, 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048567+0000", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.19e-06 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048573+0000", 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9290000000000003e-06 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048591+0000", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8179999999999999e-05 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048601+0000", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8300000000000008e-06 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059910+0000", 2024-09-11T05:51:26.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011309476000000001 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059919+0000", 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9660000000000002e-06 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060034+0000", 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011445700000000001 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14814 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631881+0000", 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52102993799999997, 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096977239999999996, 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14814 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631881+0000", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631881+0000", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631881+0000", 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.75e-07 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631883+0000", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4679999999999999e-06 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631890+0000", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8179999999999998e-06 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631900+0000", 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.006e-05 2024-09-11T05:51:26.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641380+0000", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094795230000000001 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641469+0000", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9283999999999994e-05 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641578+0000", 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109096 2024-09-11T05:51:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14815 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631908+0000", 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.521002356, 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098715209999999994, 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.174 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14815 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631908+0000", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631908+0000", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631909+0000", 2024-09-11T05:51:26.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6400000000000003e-07 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631911+0000", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2510000000000001e-06 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631918+0000", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.289e-06 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631924+0000", 2024-09-11T05:51:26.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5489999999999997e-06 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641613+0000", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096890940000000005 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641689+0000", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5764000000000002e-05 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641780+0000", 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.111e-05 2024-09-11T05:51:26.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14816 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52097911100000005, 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010070437, 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.178 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14816 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631932+0000", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4499999999999998e-07 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631934+0000", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.136e-06 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631940+0000", 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5679999999999996e-06 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631946+0000", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4349999999999996e-06 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641825+0000", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098788680000000007 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641897+0000", 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2577999999999995e-05 2024-09-11T05:51:26.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642002+0000", 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010450699999999999 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14824 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52095582500000004, 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011351063, 2024-09-11T05:51:26.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14824 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.58e-07 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631957+0000", 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4169999999999999e-06 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631965+0000", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6149999999999999e-06 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631971+0000", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7570000000000001e-06 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642045+0000", 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010074112 2024-09-11T05:51:26.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642124+0000", 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9067999999999996e-05 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643306+0000", 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011817360000000001 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14825 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 3670016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52093231299999998, 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010650197, 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14825 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631979+0000", 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2199999999999999e-07 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631981+0000", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.176e-06 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631988+0000", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4490000000000002e-06 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631994+0000", 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.507e-06 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642464+0000", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010470187000000001 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642540+0000", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6058999999999998e-05 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642628+0000", 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8397000000000001e-05 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14833 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52090948100000001, 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010835374, 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14833 2024-09-11T05:51:26.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:26.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0999999999999999e-07 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632003+0000", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0600000000000002e-06 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632011+0000", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2790000000000001e-06 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632016+0000", 2024-09-11T05:51:26.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5450000000000003e-06 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642656+0000", 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010639414999999999 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642728+0000", 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2166e-05 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642836+0000", 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010849900000000001 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14834 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52088689300000002, 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011001664, 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14834 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632026+0000", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632035+0000", 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.217e-06 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632041+0000", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5289999999999999e-06 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642862+0000", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010820689 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642935+0000", 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3579999999999997e-05 2024-09-11T05:51:26.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643025+0000", 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0070999999999999e-05 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14835 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52086165200000001, 2024-09-11T05:51:26.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011160765, 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14835 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9799999999999999e-07 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632052+0000", 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4480000000000001e-06 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632060+0000", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3259999999999996e-06 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632068+0000", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6830000000000008e-06 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643051+0000", 2024-09-11T05:51:26.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010983153000000001 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643125+0000", 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3967000000000005e-05 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643210+0000", 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4790000000000006e-05 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14844 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52083470700000001, 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011642978999999999, 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14844 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632082+0000", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632095+0000", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3454e-05 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632102+0000", 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1430000000000003e-06 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643234+0000", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011132918 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643339+0000", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104734 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643719+0000", 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000379551 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.207 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14845 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52079752099999999, 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011793567, 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14845 2024-09-11T05:51:26.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5299999999999999e-07 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632116+0000", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3039999999999999e-06 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632124+0000", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6640000000000004e-06 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632127+0000", 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.948e-06 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643806+0000", 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011678932 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643856+0000", 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0198999999999998e-05 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643907+0000", 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0067e-05 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14846 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52077939600000001, 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01202668, 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14846 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.23e-07 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632138+0000", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2559999999999999e-06 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632143+0000", 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0370000000000002e-06 2024-09-11T05:51:26.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632146+0000", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5489999999999998e-06 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644063+0000", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011917116 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644109+0000", 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5676000000000001e-05 2024-09-11T05:51:26.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644158+0000", 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8823000000000003e-05 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14847 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52076044300000002, 2024-09-11T05:51:26.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012106896000000001, 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14847 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.91e-07 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632152+0000", 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.767e-06 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632158+0000", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632162+0000", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644175+0000", 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012013835 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644214+0000", 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8216999999999997e-05 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644257+0000", 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3257999999999998e-05 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14874 3.0 3:040978a7:::rbd_data.1124ef904871.0000000000000004:a [sparse-read 3407872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.220 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52074260699999997, 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012189050999999999, 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14874 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:26.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5799999999999999e-07 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632170+0000", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0310000000000001e-06 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632176+0000", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7780000000000004e-06 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632181+0000", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3710000000000002e-06 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644274+0000", 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012093303999999999 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644315+0000", 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0683000000000003e-05 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644357+0000", 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2326000000000001e-05 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14875 3.0 3:040978a7:::rbd_data.1124ef904871.0000000000000004:a [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52071980500000004, 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012260057, 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14875 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8799999999999998e-07 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632193+0000", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1969999999999999e-06 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632200+0000", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4139999999999999e-06 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632205+0000", 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8380000000000001e-06 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644372+0000", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012167585 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644410+0000", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7462000000000001e-05 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644451+0000", 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1173e-05 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.228 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14878 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52070152800000002, 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012364757000000001, 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14878 2024-09-11T05:51:26.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:26.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5499999999999999e-07 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632211+0000", 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.328e-06 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632214+0000", 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6169999999999998e-06 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632223+0000", 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3960000000000006e-06 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644467+0000", 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012244147 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644527+0000", 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0186000000000003e-05 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644574+0000", 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6828e-05 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14879 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.52068125399999998, 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012781137, 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:26.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14879 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632230+0000", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632232+0000", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0930000000000001e-06 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632238+0000", 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2840000000000003e-06 2024-09-11T05:51:26.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632243+0000", 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5809999999999998e-06 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644660+0000", 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012417901 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644879+0000", 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021857800000000001 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645010+0000", 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013134400000000001 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:15154 3.19 3:9b307830:::rbd_data.11243bbd48bb.0000000000000039:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.25177534000000001, 2024-09-11T05:51:26.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016613869999999999, 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 15154 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901138+0000", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4329999999999998e-06 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901148+0000", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0553e-05 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901168+0000", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0360000000000002e-05 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901177+0000", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7660000000000004e-06 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901251+0000", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3504999999999995e-05 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901344+0000", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3381999999999995e-05 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.904163+0000", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028184880000000001 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917590+0000", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013427346999999999 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:26.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917644+0000", 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3659000000000002e-05 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917700+0000", 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5995000000000002e-05 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917749+0000", 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9382000000000003e-05 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.243 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:26.243 INFO:teuthology.orchestra.run.smithi165.stdout:1626 write 0xdb772a9 thru 0xdb80d50 (0x9aa8 bytes) 2024-09-11T05:51:26.243 INFO:teuthology.orchestra.run.smithi165.stdout:1628 trunc from 0xdb80d51 to 0xc1a590d 2024-09-11T05:51:26.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1629 read 0x5a4a9d thru 0x5af631 (0xab95 bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1630 trunc from 0xc1a590d to 0xe293ec2 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1631 write 0xe9b998 thru 0xea2eb5 (0x751e bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1633 read 0x8e8e1a1 thru 0x8e94bb5 (0x6a15 bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1634 write 0xd63c05b thru 0xd649027 (0xcfcd bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1636 write 0x16a7ca8 thru 0x16af6b5 (0x7a0e bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1640 read 0xe06a2e0 thru 0xe073334 (0x9055 bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1641 writesame 0x1c24268 thru 0x1c2cedc data_size 0x1(0x8c75 bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1644 write 0xd74c5bd thru 0xd75a409 (0xde4d bytes) 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.246 INFO:teuthology.orchestra.run.smithi165.stdout:1646 writesame 0xc516e66 thru 0xc51d13f data_size 0x1(0x62da bytes) 2024-09-11T05:51:26.247 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.247 INFO:teuthology.orchestra.run.smithi165.stdout:1650 write 0x90c3a1d thru 0x90cf03b (0xb61f bytes) 2024-09-11T05:51:26.247 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:26.242+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:26.248 INFO:teuthology.orchestra.run.smithi165.stdout:1651 punch from 0xb26ec31 to 0xb2773e7, (0x87b6 bytes) 2024-09-11T05:51:26.264 INFO:teuthology.orchestra.run.smithi165.stdout:1652 trunc from 0xe293ec2 to 0x10d90e3 2024-09-11T05:51:26.306 INFO:teuthology.orchestra.run.smithi165.stdout:1654 writesame 0x8b68ab7 thru 0x8b6e6fe data_size 0x4(0x5c48 bytes) 2024-09-11T05:51:26.306 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.313 INFO:teuthology.orchestra.run.smithi165.stdout:1656 trunc from 0x8b6e6ff to 0x381ed9b 2024-09-11T05:51:26.321 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.334 INFO:teuthology.orchestra.run.smithi165.stdout:1659 write 0x66f84a6 thru 0x67056d8 (0xd233 bytes) 2024-09-11T05:51:26.341 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.341 INFO:teuthology.orchestra.run.smithi165.stdout:1661 writesame 0x505ca3 thru 0x50c77c data_size 0x2f(0x6ada bytes) 2024-09-11T05:51:26.345 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:26.342+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:26.358 INFO:teuthology.orchestra.run.smithi165.stdout:1663 writesame 0x93726c8 thru 0x937db59 data_size 0x1d(0xb492 bytes) 2024-09-11T05:51:26.358 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.358 INFO:teuthology.orchestra.run.smithi165.stdout:1666 read 0x8194721 thru 0x8199ed2 (0x57b2 bytes) 2024-09-11T05:51:26.358 INFO:teuthology.orchestra.run.smithi165.stdout:1668 punch from 0x89978b to 0x8a5101, (0xb976 bytes) 2024-09-11T05:51:26.359 INFO:teuthology.orchestra.run.smithi165.stdout:1669 read 0x7554a6d thru 0x75640e2 (0xf676 bytes) 2024-09-11T05:51:26.359 INFO:teuthology.orchestra.run.smithi165.stdout:1671 writesame 0x2be918 thru 0x2bed6a data_size 0x3(0x453 bytes) 2024-09-11T05:51:26.367 INFO:teuthology.orchestra.run.smithi165.stdout:1672 trunc from 0x937db5a to 0xd96d071 2024-09-11T05:51:26.377 INFO:teuthology.orchestra.run.smithi165.stdout:1673 punch from 0x2197689 to 0x21a69d5, (0xf34c bytes) 2024-09-11T05:51:26.378 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:26.378 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:26.379 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:26.379 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:26.379 INFO:teuthology.orchestra.run.smithi165.stdout:1675 trunc from 0xd96d071 to 0xecd055c 2024-09-11T05:51:26.389 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.389 INFO:teuthology.orchestra.run.smithi165.stdout:1678 write 0xdd10d7b thru 0xdd11158 (0x3de bytes) 2024-09-11T05:51:26.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:26.395 INFO:teuthology.orchestra.run.smithi165.stdout:1680 writesame 0x1729020 thru 0x1737bd9 data_size 0x1a(0xebba bytes) 2024-09-11T05:51:26.400 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.400 INFO:teuthology.orchestra.run.smithi165.stdout:1683 read 0x611dab thru 0x61d58f (0xb7e5 bytes) 2024-09-11T05:51:26.401 INFO:teuthology.orchestra.run.smithi165.stdout:1686 read 0x5bfc707 thru 0x5c032e5 (0x6bdf bytes) 2024-09-11T05:51:26.402 INFO:teuthology.orchestra.run.smithi165.stdout:1687 write 0xc6eb4d9 thru 0xc6f006a (0x4b92 bytes) 2024-09-11T05:51:26.409 INFO:teuthology.orchestra.run.smithi165.stdout:1688 clone 9 order 20 su 16384 sc 3 2024-09-11T05:51:26.447 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:26.446+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:26.524 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:26.524 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:26.524 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:26.524 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:26.524 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:26.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:26.587 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:26.586+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:26.647 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:26.646+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:26.648 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T05:51:26.658 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:51:26.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.510778065, 2024-09-11T05:51:26.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:26.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:26.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6053 3.0 e33/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.419313699, 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019874799999999998, 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303401+0000", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303403+0000", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.378e-06 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303414+0000", 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.155e-05 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303437+0000", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2342000000000001e-05 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303446+0000", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6760000000000002e-06 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303513+0000", 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6431000000000001e-05 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.303536+0000", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3258999999999999e-05 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305349+0000", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018126640000000001 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.305389+0000", 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0179999999999998e-05 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.730 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6060 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'29, mlcod=33'27)", 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.40846743, 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024674810000000001, 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:26.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4099999999999997e-07 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314255+0000", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314269+0000", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4177e-05 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314277+0000", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6279999999999996e-06 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314341+0000", 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3539000000000003e-05 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314363+0000", 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2272e-05 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316606+0000", 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002243629 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316715+0000", 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108576 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.407788333999999, 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.735 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:26.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6091 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'17, mlcod=33'15)", 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.365628367999999, 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019923990000000002, 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357087+0000", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357088+0000", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3829999999999999e-06 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357127+0000", 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8819999999999998e-05 2024-09-11T05:51:26.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357144+0000", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7027000000000001e-05 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357153+0000", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5810000000000003e-06 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357213+0000", 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9959e-05 2024-09-11T05:51:26.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.357231+0000", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7983e-05 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359040+0000", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018086090000000001 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.359079+0000", 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9036999999999999e-05 2024-09-11T05:51:26.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.313246297999999, 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.041499013999999, 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.748 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:26.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6343 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'23, mlcod=33'21)", 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.9210841389999995, 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001815929, 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801631+0000", 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801642+0000", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1019000000000001e-05 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801673+0000", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1349000000000001e-05 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801694+0000", 2024-09-11T05:51:26.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1067999999999999e-05 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801706+0000", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1731e-05 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801771+0000", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4613e-05 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.801789+0000", 2024-09-11T05:51:26.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8839e-05 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803393+0000", 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016039299999999999 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.803447+0000", 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3380000000000001e-05 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.9174800400000009, 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.760 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6354 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'25, mlcod=33'23)", 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.9061068680000002, 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026120029999999999, 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816609+0000", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816617+0000", 2024-09-11T05:51:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7680000000000008e-06 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816630+0000", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816637+0000", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816690+0000", 2024-09-11T05:51:26.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2533e-05 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816706+0000", 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6407000000000001e-05 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819170+0000", 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024645309999999998 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819220+0000", 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9595999999999997e-05 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.8424878640000006, 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:26.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:26.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:26.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6672 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'39, mlcod=33'37)", 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:26.769 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.4239342659999998, 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022554760000000002, 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:26.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298783+0000", 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.362e-06 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298845+0000", 2024-09-11T05:51:26.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1544999999999994e-05 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298878+0000", 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3642000000000001e-05 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298895+0000", 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6750000000000001e-05 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299066+0000", 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017087600000000001 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299124+0000", 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8213e-05 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.300980+0000", 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00185586 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.301036+0000", 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6227999999999998e-05 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3685127440000002, 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.774 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:26.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:26.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6924 3.0 e34/23 3:06ec5f12:::rbd_data.11249b359231.0000000000000026:head v 34'25, mlcod=34'23)", 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:26.778 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.0212820970000003, 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019233869999999999, 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:26.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701433+0000", 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2e-07 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701440+0000", 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.314e-06 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701448+0000", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1939999999999998e-06 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701474+0000", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6336999999999999e-05 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701570+0000", 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5503999999999999e-05 2024-09-11T05:51:26.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.701588+0000", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8510000000000001e-05 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703303+0000", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017153380000000001 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.703356+0000", 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2769999999999998e-05 2024-09-11T05:51:26.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.8758834800000006, 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7116 3.9 e34/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 34'35, mlcod=34'33)", 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.7156555030000007, 2024-09-11T05:51:26.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00371405, 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:26.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007061+0000", 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5090000000000001e-06 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007072+0000", 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1399e-05 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007100+0000", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7531000000000001e-05 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007114+0000", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4378e-05 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007340+0000", 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022568499999999999 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007387+0000", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7194000000000002e-05 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010717+0000", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033296369999999999 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010773+0000", 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6716999999999999e-05 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7119 3.4 e34/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 34'49, mlcod=34'47)", 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.7144497249999997, 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034162110000000002, 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008267+0000", 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.294e-06 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008276+0000", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5820000000000008e-06 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008290+0000", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4005000000000001e-05 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008298+0000", 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3189999999999995e-06 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008368+0000", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0055999999999995e-05 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008396+0000", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7563999999999999e-05 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011628+0000", 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032314969999999998 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011681+0000", 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3894000000000003e-05 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12734 3.0 e38/23 3:06e59eda:::rbd_data.1124518b2c12.000000000000002b:head v 38'41, mlcod=37'39)", 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9124397220000002, 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033376479999999999, 2024-09-11T05:51:26.796 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810277+0000", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810314+0000", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7370999999999998e-05 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810336+0000", 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2129000000000001e-05 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810347+0000", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0465999999999999e-05 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810418+0000", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099000000000005e-05 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810443+0000", 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4825999999999999e-05 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813553+0000", 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031099790000000001 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813613+0000", 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0307999999999999e-05 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.800 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12847 3.0 e38/23 3:06e59eda:::rbd_data.1124518b2c12.000000000000002b:head v 38'44, mlcod=38'42)", 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:23.070907+0000", 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.6518078460000001, 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001837389, 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.070907+0000", 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.070907+0000", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.070910+0000", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9000000000000002e-06 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.070969+0000", 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9026e-05 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.071011+0000", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1993000000000003e-05 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.071032+0000", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1466000000000001e-05 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.071191+0000", 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000158083 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.071240+0000", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9712000000000001e-05 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.072663+0000", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014225069999999999 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:23.072744+0000", 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1701999999999998e-05 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.804 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15206 3.0 e41/23 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:head v 41'65, mlcod=41'63)", 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.71873949100000001, 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034631129999999999, 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003977+0000", 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003986+0000", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004008+0000", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1679e-05 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004018+0000", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5210000000000008e-06 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004105+0000", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7371000000000001e-05 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004128+0000", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2682999999999999e-05 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007395+0000", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032676129999999999 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:26.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007439+0000", 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3306999999999997e-05 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.809 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:26.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:26.810 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone8 from 0xecd055c (overlap 0xa84ac1) to 0x86a234 2024-09-11T05:51:26.811 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:26.810+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:26.853 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-09-11T05:51:26.856 INFO:teuthology.orchestra.run.smithi165.stdout:1689 read 0x3b2f487 thru 0x3b3060e (0x1188 bytes) 2024-09-11T05:51:26.857 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.857 INFO:teuthology.orchestra.run.smithi165.stdout:1691 write 0x61908ea thru 0x6197701 (0x6e18 bytes) 2024-09-11T05:51:26.875 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.875 INFO:teuthology.orchestra.run.smithi165.stdout:1693 read 0x4c63314 thru 0x4c6be06 (0x8af3 bytes) 2024-09-11T05:51:26.876 INFO:teuthology.orchestra.run.smithi165.stdout:1697 read 0x514eacf thru 0x515cdb1 (0xe2e3 bytes) 2024-09-11T05:51:26.876 INFO:teuthology.orchestra.run.smithi165.stdout:1698 write 0x9d0eb2e thru 0x9d1a1bd (0xb690 bytes) 2024-09-11T05:51:26.897 INFO:teuthology.orchestra.run.smithi165.stdout:1700 trunc from 0xecd055c to 0x69677be 2024-09-11T05:51:26.911 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:26.910+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:26.930 INFO:teuthology.orchestra.run.smithi165.stdout:1701 read 0x10ba312 thru 0x10c95a7 (0xf296 bytes) 2024-09-11T05:51:26.933 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:26.933 INFO:teuthology.orchestra.run.smithi165.stdout:1704 write 0x14296e0 thru 0x14306e8 (0x7009 bytes) 2024-09-11T05:51:26.968 INFO:teuthology.orchestra.run.smithi165.stdout:1705 writesame 0x88ce7aa thru 0x88d1319 data_size 0x14(0x2b70 bytes) 2024-09-11T05:51:26.968 INFO:teuthology.orchestra.run.smithi165.stdout:1706 read 0x5650d52 thru 0x5651d88 (0x1037 bytes) 2024-09-11T05:51:26.968 INFO:teuthology.orchestra.run.smithi165.stdout:1707 write 0x4618897 thru 0x461eb0d (0x6277 bytes) 2024-09-11T05:51:26.976 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:26.976 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:26.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:15941 3.8 3:16a968d8:::rbd_data.1124d6faf06d.0000000000000047:head [call rbd.sparse_copyup in=24b,set-alloc-hint object_size 1048576 write_size 1048576,write 376832~11022 in=11022b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.977728+0000", 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.000579063, 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061914299999999995, 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 15941 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977728+0000", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977728+0000", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977731+0000", 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3479999999999998e-06 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977742+0000", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1950000000000001e-05 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977759+0000", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6611000000000001e-05 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977766+0000", 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0480000000000001e-06 2024-09-11T05:51:26.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.977853+0000", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7225000000000006e-05 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.978064+0000", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021020199999999999 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.978355+0000", 2024-09-11T05:51:26.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029149299999999998 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:26.981 INFO:teuthology.orchestra.run.smithi165.stdout:1708 write 0x5df23dd thru 0x5dfb0d4 (0x8cf8 bytes) 2024-09-11T05:51:26.990 INFO:teuthology.orchestra.run.smithi165.stdout:1710 write 0x73e686d thru 0x73e7321 (0xab5 bytes) 2024-09-11T05:51:26.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:27.005 INFO:teuthology.orchestra.run.smithi165.stdout:1712 trunc from 0x88d131a to 0x302591 2024-09-11T05:51:27.013 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:27.014+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:27.042 INFO:teuthology.orchestra.run.smithi165.stdout:1715 punch from 0x1fd37d to 0x207b76, (0xa7f9 bytes) 2024-09-11T05:51:27.042 INFO:teuthology.orchestra.run.smithi165.stdout:1716 read 0x11230a thru 0x115c7e (0x3975 bytes) 2024-09-11T05:51:27.055 INFO:teuthology.orchestra.run.smithi165.stdout:1717 write 0x31515dc thru 0x315a7b5 (0x91da bytes) 2024-09-11T05:51:27.067 INFO:teuthology.orchestra.run.smithi165.stdout:1719 write 0x144d307 thru 0x144f1bb (0x1eb5 bytes) 2024-09-11T05:51:27.078 INFO:teuthology.orchestra.run.smithi165.stdout:1720 trunc from 0x315a7b6 to 0x5637385 2024-09-11T05:51:27.086 INFO:teuthology.orchestra.run.smithi165.stdout:1721 read 0x1a7b754 thru 0x1a7c11b (0x9c8 bytes) 2024-09-11T05:51:27.087 INFO:teuthology.orchestra.run.smithi165.stdout:1724 punch from 0x1db928f to 0x1dba607, (0x1378 bytes) 2024-09-11T05:51:27.105 INFO:teuthology.orchestra.run.smithi165.stdout:1725 writesame 0xe132d75 thru 0xe137cbb data_size 0x63(0x4f47 bytes) 2024-09-11T05:51:27.106 INFO:teuthology.orchestra.run.smithi165.stdout:1727 writesame 0x37859e1 thru 0x3793d48 data_size 0x4c(0xe368 bytes) 2024-09-11T05:51:27.113 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:27.114+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:27.121 INFO:teuthology.orchestra.run.smithi165.stdout:1728 trunc from 0xe137cbc to 0xc62f1d5 2024-09-11T05:51:27.127 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:27.127 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:27.127 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:27.127 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:27.127 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:27.131 INFO:teuthology.orchestra.run.smithi165.stdout:1729 read 0x4ee4940 thru 0x4eef655 (0xad16 bytes) 2024-09-11T05:51:27.131 INFO:teuthology.orchestra.run.smithi165.stdout:1730 write 0x3f90f42 thru 0x3f9c5cb (0xb68a bytes) 2024-09-11T05:51:27.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:27.142 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.142 INFO:teuthology.orchestra.run.smithi165.stdout:1732 write 0x8c183fd thru 0x8c275c4 (0xf1c8 bytes) 2024-09-11T05:51:27.153 INFO:teuthology.orchestra.run.smithi165.stdout:1733 write 0xa7c14ae thru 0xa7d1160 (0xfcb3 bytes) 2024-09-11T05:51:27.163 INFO:teuthology.orchestra.run.smithi165.stdout:1734 write 0x1c9c3d1 thru 0x1c9dcb7 (0x18e7 bytes) 2024-09-11T05:51:27.168 INFO:teuthology.orchestra.run.smithi165.stdout:1735 read 0x4fd71e3 thru 0x4fddd71 (0x6b8f bytes) 2024-09-11T05:51:27.168 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.171 INFO:teuthology.orchestra.run.smithi165.stdout:1737 trunc from 0xc62f1d5 to 0xe66a0f9 2024-09-11T05:51:27.180 INFO:teuthology.orchestra.run.smithi165.stdout:1738 read 0x80b516f thru 0x80c4a4f (0xf8e1 bytes) 2024-09-11T05:51:27.181 INFO:teuthology.orchestra.run.smithi165.stdout:1739 punch from 0xbce8f32 to 0xbceb65b, (0x2729 bytes) 2024-09-11T05:51:27.181 INFO:teuthology.orchestra.run.smithi165.stdout:1741 writesame 0xd8795b3 thru 0xd87fd6b data_size 0x3(0x67b9 bytes) 2024-09-11T05:51:27.193 INFO:teuthology.orchestra.run.smithi165.stdout:1742 read 0xb53419c thru 0xb5383a3 (0x4208 bytes) 2024-09-11T05:51:27.193 INFO:teuthology.orchestra.run.smithi165.stdout:1743 punch from 0xb625caa to 0xb62eb69, (0x8ebf bytes) 2024-09-11T05:51:27.206 INFO:teuthology.orchestra.run.smithi165.stdout:1746 trunc from 0xe66a0f9 to 0x80dc0bd 2024-09-11T05:51:27.214 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:27.214+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:27.230 INFO:teuthology.orchestra.run.smithi165.stdout:1747 read 0x3bbbfff thru 0x3bca32c (0xe32e bytes) 2024-09-11T05:51:27.230 INFO:teuthology.orchestra.run.smithi165.stdout:1748 read 0x4880802 thru 0x4888ecc (0x86cb bytes) 2024-09-11T05:51:27.230 INFO:teuthology.orchestra.run.smithi165.stdout:1749 read 0x1dbd6e7 thru 0x1dc1a70 (0x438a bytes) 2024-09-11T05:51:27.230 INFO:teuthology.orchestra.run.smithi165.stdout:1750 write 0x38fb36c thru 0x3908018 (0xccad bytes) 2024-09-11T05:51:27.243 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.244 INFO:teuthology.orchestra.run.smithi165.stdout:1752 read 0x51348ac thru 0x513f6b6 (0xae0b bytes) 2024-09-11T05:51:27.244 INFO:teuthology.orchestra.run.smithi165.stdout:1753 read 0x41b5745 thru 0x41c523a (0xfaf6 bytes) 2024-09-11T05:51:27.245 INFO:teuthology.orchestra.run.smithi165.stdout:1754 clone 10 order 18 su 16384 sc 10 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:881 3.b e28/23 ondisk, result = 0)", 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.826838227, 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010956011, 2024-09-11T05:51:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438670+0000", 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438671+0000", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438675+0000", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1910000000000002e-06 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438685+0000", 2024-09-11T05:51:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0001e-05 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.438692+0000", 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3830000000000003e-06 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449422+0000", 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010729394 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449429+0000", 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6599999999999995e-06 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:11.449626+0000", 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019679800000000001 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1198 3.8 3:15f20ad5:::rbd_data.112475afb12f.00000000000000b6:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.115608194, 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013528144000000001, 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1198 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149901+0000", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.172e-06 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149915+0000", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3801000000000001e-05 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149929+0000", 2024-09-11T05:51:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3624e-05 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149935+0000", 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149998+0000", 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3065999999999996e-05 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.150144+0000", 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014578400000000001 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.152779+0000", 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002635025 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163265+0000", 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010485758 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163327+0000", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2415000000000005e-05 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163372+0000", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4331e-05 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163428+0000", 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.664e-05 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6095 3.b 3:d107d222:::rbd_data.11249b359231.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.903027626, 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012675989, 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6095 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362482+0000", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099e-06 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362488+0000", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4869999999999999e-06 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362502+0000", 2024-09-11T05:51:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3407999999999999e-05 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362510+0000", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1799999999999996e-06 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362567+0000", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7538999999999999e-05 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362653+0000", 2024-09-11T05:51:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6051999999999995e-05 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364892+0000", 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022386630000000001 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.374988+0000", 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010095860999999999 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375041+0000", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2785999999999999e-05 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375105+0000", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4715999999999995e-05 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375157+0000", 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1198e-05 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12375 3.b 3:d592f06e:::rbd_data.1124518b2c12.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 3542259~8433] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1800534899999997, 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010649456999999999, 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12375 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085455+0000", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.085456+0000", 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999991e-07 2024-09-11T05:51:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086709+0000", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012536859999999999 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086732+0000", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2459999999999998e-05 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086742+0000", 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6330000000000008e-06 2024-09-11T05:51:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086807+0000", 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5780999999999999e-05 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.086935+0000", 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012802700000000001 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.087312+0000", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000376948 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.095385+0000", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080731789999999998 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096024+0000", 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063872600000000005 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096068+0000", 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3427999999999998e-05 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.096104+0000", 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6628999999999998e-05 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12436 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0557442909999999, 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095948849999999992, 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12436 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209764+0000", 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209766+0000", 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5170000000000002e-06 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209796+0000", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9689999999999999e-05 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209831+0000", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4913000000000003e-05 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209855+0000", 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3643e-05 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.210745+0000", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089045300000000003 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.210963+0000", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218068 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.213370+0000", 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002406975 2024-09-11T05:51:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.215267+0000", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018967910000000001 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219268+0000", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040012959999999997 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219307+0000", 2024-09-11T05:51:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9042999999999999e-05 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.219359+0000", 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1496000000000003e-05 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12439 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.0556004919999999, 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011043397999999999, 2024-09-11T05:51:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12439 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209908+0000", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209909+0000", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6729999999999999e-06 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209919+0000", 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1570000000000006e-06 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209949+0000", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0029999999999999e-05 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.209965+0000", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6290000000000002e-05 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214405+0000", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044401609999999998 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214509+0000", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010385700000000001 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.214956+0000", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044748200000000002 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.215653+0000", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069695599999999999 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220661+0000", 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050078839999999998 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220831+0000", 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017013600000000001 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.220951+0000", 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119772 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12762 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4009930580000001, 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094195340000000002, 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12762 2024-09-11T05:51:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864515+0000", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864516+0000", 2024-09-11T05:51:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864524+0000", 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2330000000000006e-06 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864532+0000", 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0059999999999992e-06 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864537+0000", 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5300000000000004e-06 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867222+0000", 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002684475 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867266+0000", 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4604999999999997e-05 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867445+0000", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017825499999999999 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869566+0000", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021212119999999999 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873837+0000", 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042710760000000004 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873892+0000", 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4981000000000001e-05 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.873935+0000", 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2710999999999998e-05 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:12763 3.7 3:e65ff244:::rbd_object_map.1124518b2c12:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4009626710000003, 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098476699999999993, 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 12763 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864546+0000", 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0500000000000004e-07 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864549+0000", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8389999999999998e-06 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864556+0000", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1949999999999997e-06 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.864561+0000", 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3240000000000002e-06 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867714+0000", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031524370000000001 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867758+0000", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3903000000000003e-05 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.867934+0000", 2024-09-11T05:51:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017616499999999999 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.869572+0000", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016385309999999999 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874165+0000", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045921989999999999 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874236+0000", 2024-09-11T05:51:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1491999999999999e-05 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.874393+0000", 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015708000000000001 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14829 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 3407872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631140+0000", 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6343685429999999, 2024-09-11T05:51:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095296449999999998, 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14829 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631140+0000", 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631140+0000", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631140+0000", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8700000000000001e-07 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631142+0000", 2024-09-11T05:51:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.84e-06 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631148+0000", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0140000000000004e-06 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631153+0000", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5639999999999998e-06 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640507+0000", 2024-09-11T05:51:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093549159999999996 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640588+0000", 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0649999999999995e-05 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640669+0000", 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1273999999999995e-05 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14840 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631215+0000", 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.634292769, 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096703199999999996, 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14840 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631215+0000", 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631215+0000", 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631216+0000", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.58e-07 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631218+0000", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1009999999999999e-06 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631224+0000", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3890000000000002e-06 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631229+0000", 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8670000000000002e-06 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640698+0000", 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094683059999999993 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640780+0000", 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2311000000000005e-05 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640886+0000", 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105888 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14841 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.634272494, 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013476685, 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14841 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6600000000000002e-07 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631238+0000", 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9120000000000001e-06 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631244+0000", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2809999999999997e-06 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631249+0000", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6550000000000003e-06 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640915+0000", 2024-09-11T05:51:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096659480000000006 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640996+0000", 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1108999999999999e-05 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644712+0000", 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037164139999999999 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14842 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6342332449999999, 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013741382999999999, 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14842 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631276+0000", 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:51:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631280+0000", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9470000000000002e-06 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631287+0000", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0450000000000003e-06 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631293+0000", 2024-09-11T05:51:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.006e-06 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644746+0000", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013453074000000001 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644876+0000", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012993999999999999 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645016+0000", 2024-09-11T05:51:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014087299999999999 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14843 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6342081849999999, 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014071284999999999, 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14843 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631301+0000", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2600000000000002e-07 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631303+0000", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.962e-06 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631309+0000", 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0650000000000004e-06 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631313+0000", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8380000000000001e-06 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645202+0000", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013888282 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645283+0000", 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0793999999999999e-05 2024-09-11T05:51:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645371+0000", 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8818000000000002e-05 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14863 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6341881730000001, 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014249672, 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14863 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2e-07 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631322+0000", 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.968e-06 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631331+0000", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0949999999999996e-06 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631336+0000", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0969999999999996e-06 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645401+0000", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014064933 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645484+0000", 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3325999999999995e-05 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645570+0000", 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5833000000000003e-05 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14864 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4194304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6341658379999999, 2024-09-11T05:51:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014417572, 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14864 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631343+0000", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6699999999999999e-07 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631365+0000", 2024-09-11T05:51:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2541e-05 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631373+0000", 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2989999999999999e-06 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631378+0000", 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645599+0000", 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01422167 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645674+0000", 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4779999999999999e-05 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645760+0000", 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5867999999999998e-05 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14865 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.634120432, 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014562083, 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14865 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0500000000000004e-07 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631390+0000", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1090000000000001e-06 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631398+0000", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0829999999999998e-06 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631403+0000", 2024-09-11T05:51:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1680000000000003e-06 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645789+0000", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014386144 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645863+0000", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3924000000000006e-05 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645950+0000", 2024-09-11T05:51:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7150000000000004e-05 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14866 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4980736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6340989399999999, 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014728692, 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14866 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631410+0000", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.27e-07 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631412+0000", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1059999999999998e-06 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631420+0000", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631425+0000", 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9649999999999999e-06 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645979+0000", 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014554015999999999 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646051+0000", 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2631000000000003e-05 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646138+0000", 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6828000000000003e-05 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14876 3.b 3:d11944fd:::rbd_data.1124ef904871.0000000000000005:a [sparse-read 3407872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.634077054, 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014895617, 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14876 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631432+0000", 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.77e-07 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631434+0000", 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9290000000000001e-06 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631440+0000", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2670000000000003e-06 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631445+0000", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7389999999999999e-06 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646167+0000", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01472234 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646240+0000", 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3596000000000005e-05 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646327+0000", 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6368999999999999e-05 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14877 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6340557279999999, 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015071222, 2024-09-11T05:51:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14877 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1399999999999997e-07 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631455+0000", 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1619999999999998e-06 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631462+0000", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1810000000000004e-06 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631476+0000", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3263e-05 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646359+0000", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014883601 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646435+0000", 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6056999999999994e-05 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646524+0000", 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8543999999999998e-05 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:15330 3.7 3:e3450b35:::rbd_data.11243bbd48bb.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.06597184, 2024-09-11T05:51:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018829538, 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 15330 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199537+0000", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-07 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199542+0000", 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6970000000000001e-06 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199553+0000", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0831e-05 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199560+0000", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099999999999997e-06 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199598+0000", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7697999999999997e-05 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199639+0000", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1863000000000002e-05 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.202516+0000", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028767580000000001 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218204+0000", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015687326000000001 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218230+0000", 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6387e-05 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218316+0000", 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6227e-05 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218366+0000", 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9795e-05 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:27.365 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:27.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:27.366 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:27.366+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:27.468 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:27.466+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:27.497 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:27.497 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:27.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:27.569 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:27.570+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:27.628 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:27.628 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:27.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:27.663 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T05:51:27.684 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:27.686+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:27.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-11T05:51:27.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:27.769 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone9 from 0x80dc0bd (overlap 0x302591) to 0x13423a 2024-09-11T05:51:27.811 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:27.810+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:27.820 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:27.820 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:27.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:27.820 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11388 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3145728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.099574+0000", 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.722159167, 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013003689000000001, 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11388 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.821 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099574+0000", 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099574+0000", 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099575+0000", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3060000000000001e-06 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099580+0000", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2599999999999996e-06 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099597+0000", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6713000000000001e-05 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099611+0000", 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3567000000000001e-05 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.112125+0000", 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012514559999999999 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.112360+0000", 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000234668 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.112577+0000", 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000217615 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11390 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3211264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.099697+0000", 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7220362849999997, 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013390467, 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.825 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11390 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099697+0000", 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099697+0000", 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099698+0000", 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.15e-06 2024-09-11T05:51:27.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099703+0000", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0270000000000003e-06 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099722+0000", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9712999999999999e-05 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099736+0000", 2024-09-11T05:51:27.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3893000000000001e-05 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.112681+0000", 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012945095 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.112899+0000", 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000218013 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113087+0000", 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018757600000000001 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11391 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.099753+0000", 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7219795590000002, 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013971162000000001, 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11391 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099753+0000", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099753+0000", 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099754+0000", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.113e-06 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099759+0000", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9370000000000003e-06 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099776+0000", 2024-09-11T05:51:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7073000000000001e-05 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099790+0000", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.332e-05 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113270+0000", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013480343000000001 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113509+0000", 2024-09-11T05:51:27.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023881099999999999 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113724+0000", 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215565 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11393 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3342336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.099807+0000", 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.721925776, 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014363396, 2024-09-11T05:51:27.834 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11393 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099807+0000", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099807+0000", 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099808+0000", 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.083e-06 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099813+0000", 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.147e-06 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099831+0000", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7252999999999999e-05 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.099844+0000", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3183000000000001e-05 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113808+0000", 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013964496 2024-09-11T05:51:27.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.113969+0000", 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016125499999999999 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114170+0000", 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200979 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11399 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1523712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100026+0000", 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.721707275, 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014542218000000001, 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11399 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100026+0000", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100026+0000", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100027+0000", 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1999999999999999e-06 2024-09-11T05:51:27.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100036+0000", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1300000000000007e-06 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100053+0000", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7257000000000001e-05 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100072+0000", 2024-09-11T05:51:27.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8706999999999999e-05 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114248+0000", 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01417652 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114443+0000", 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019481500000000001 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114568+0000", 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012458900000000001 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11401 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100146+0000", 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7215869689999996, 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014718732999999999, 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.843 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11401 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100146+0000", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100146+0000", 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100147+0000", 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1370000000000001e-06 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100152+0000", 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1259999999999997e-06 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100169+0000", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6569999999999999e-05 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100182+0000", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3380000000000001e-05 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114611+0000", 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014429186 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114758+0000", 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014622800000000001 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114865+0000", 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107106 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11403 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1654784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.847 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100257+0000", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7214753859999998, 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014929334000000001, 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11403 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100257+0000", 2024-09-11T05:51:27.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100257+0000", 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100259+0000", 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100264+0000", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6590000000000004e-06 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100281+0000", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7184e-05 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100295+0000", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3499999999999999e-05 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.114941+0000", 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014645927 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115076+0000", 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013471199999999999 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115187+0000", 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111292 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11416 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3162112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100487+0000", 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7212459750000004, 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014985515, 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11416 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100487+0000", 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100487+0000", 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100488+0000", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3769999999999999e-06 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100495+0000", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5699999999999998e-06 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100512+0000", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7580000000000001e-05 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100526+0000", 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3366e-05 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115258+0000", 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014732719999999999 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115370+0000", 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011197299999999999 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115472+0000", 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010192900000000001 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11417 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3227648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100543+0000", 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7211901750000003, 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015219722, 2024-09-11T05:51:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11417 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100543+0000", 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100543+0000", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100544+0000", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1999999999999999e-06 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100549+0000", 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1660000000000002e-06 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100571+0000", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2263999999999999e-05 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100585+0000", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3286e-05 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115542+0000", 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014957270999999999 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115652+0000", 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110079 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115762+0000", 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110456 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11419 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3293184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.860 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100601+0000", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.721131379, 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015450234, 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11419 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100601+0000", 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100601+0000", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100603+0000", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2500000000000001e-06 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100608+0000", 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.237e-06 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100626+0000", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7546000000000001e-05 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100638+0000", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-05 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115841+0000", 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015203118999999999 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.115948+0000", 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106956 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116052+0000", 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103526 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11457 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100654+0000", 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7210784300000004, 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015684068999999998, 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11457 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100654+0000", 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100654+0000", 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100656+0000", 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2640000000000001e-06 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100661+0000", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.13e-06 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100682+0000", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1005999999999998e-05 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100699+0000", 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7407000000000001e-05 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116126+0000", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015426741000000001 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116234+0000", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107666 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116338+0000", 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104855 2024-09-11T05:51:27.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11458 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1540096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100717+0000", 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.721016283, 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015906567, 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11458 2024-09-11T05:51:27.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100717+0000", 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100717+0000", 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100718+0000", 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100723+0000", 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1000000000000003e-06 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100741+0000", 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8453000000000001e-05 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100755+0000", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3719e-05 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116413+0000", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015657919999999999 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116522+0000", 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109202 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116623+0000", 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100926 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11460 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1605632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100771+0000", 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7209613600000004, 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016134441999999999, 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11460 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100771+0000", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100771+0000", 2024-09-11T05:51:27.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100773+0000", 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.246e-06 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100778+0000", 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4070000000000001e-06 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100794+0000", 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6235999999999998e-05 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100808+0000", 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3210000000000001e-05 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116696+0000", 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015887953999999999 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116805+0000", 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010905599999999999 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116906+0000", 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101333 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.877 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11462 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 1671168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.100824+0000", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7209089009999996, 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016378186999999999, 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11462 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100824+0000", 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100824+0000", 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100825+0000", 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3939999999999999e-06 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100831+0000", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1660000000000002e-06 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100847+0000", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.679e-05 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.100866+0000", 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8587e-05 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.116984+0000", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016117647999999998 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117098+0000", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114459 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117202+0000", 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104143 2024-09-11T05:51:27.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11465 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3178496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.101006+0000", 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7207273089999999, 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016484193000000001, 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.882 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11465 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101006+0000", 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101006+0000", 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101007+0000", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101012+0000", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4260000000000002e-06 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101030+0000", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7453e-05 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101044+0000", 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3967e-05 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117278+0000", 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016234555000000001 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117388+0000", 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109962 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117490+0000", 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101456 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11467 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3244032~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.101142+0000", 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7205906730000002, 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016629860999999999, 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11467 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101142+0000", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101142+0000", 2024-09-11T05:51:27.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101143+0000", 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1400000000000001e-06 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101149+0000", 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4399999999999996e-06 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101166+0000", 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6935999999999998e-05 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101179+0000", 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2996999999999999e-05 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117563+0000", 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016384777 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117672+0000", 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108436 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117772+0000", 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100135 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.890 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11468 3.9 3:938b1af5:::rbd_data.11249b359231.0000000000000001:7 [read 3309568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.101199+0000", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.7205338269999997, 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016834709999999999, 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11468 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101199+0000", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101199+0000", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101200+0000", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101207+0000", 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7299999999999999e-06 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101227+0000", 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0400000000000001e-05 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.101247+0000", 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117819+0000", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016572518000000001 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.117930+0000", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011069300000000001 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.118034+0000", 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103639 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.894 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11636 3.1a 3:5ee2e31d:::rbd_data.1124518b2c12.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.6045691719999997, 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023307497, 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.895 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11636 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217167+0000", 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0000000000000001e-06 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221731+0000", 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045638420000000002 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221766+0000", 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221782+0000", 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5899000000000001e-05 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221933+0000", 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015070599999999999 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.222312+0000", 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037949700000000002 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.224398+0000", 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00208618 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.237521+0000", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013122642 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240180+0000", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002659509 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240337+0000", 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240471+0000", 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013454299999999999 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12094 3.14 3:29cbd9d3:::rbd_data.1124518b2c12.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 3104699~59461] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.0981394509999998, 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021958854, 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12094 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723596+0000", 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0800000000000002e-06 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728015+0000", 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004418684 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728048+0000", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3299000000000001e-05 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728064+0000", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5583e-05 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728187+0000", 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122583 2024-09-11T05:51:27.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728429+0000", 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024270499999999999 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.729310+0000", 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088068599999999999 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:27.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.743989+0000", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01467861 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745369+0000", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001379983 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745475+0000", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745552+0000", 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7391999999999994e-05 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:14882 3.9 3:92ca3bb0:::rbd_data.1124ef904871.000000000000000c:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:25.638981+0000", 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.1827522930000001, 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013314372999999999, 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:27.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 14882 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.638981+0000", 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:27.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.638981+0000", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.638982+0000", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.638987+0000", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1730000000000003e-06 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:27.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.639006+0000", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8977000000000001e-05 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.639021+0000", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4859e-05 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.652008+0000", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012987113 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:27.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.652139+0000", 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013135600000000001 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.652295+0000", 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000155735 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:27.911 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:27.912 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:27.913 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:27.913 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-09-11T05:51:27.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:27.914 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:27.914+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:27.926 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:27.926 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:27.931 INFO:teuthology.orchestra.run.smithi165.stdout:1756 write 0x3745296 thru 0x3746175 (0xee0 bytes) 2024-09-11T05:51:27.945 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:27.945 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:27.956 INFO:teuthology.orchestra.run.smithi165.stdout:1757 trunc from 0x80dc0bd to 0x28f2025 2024-09-11T05:51:27.962 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:27.963 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:27.983 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:27.983 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:27.994 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.994 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.994 INFO:teuthology.orchestra.run.smithi165.stdout:1762 punch from 0xfcb12c to 0xfd2724, (0x75f8 bytes) 2024-09-11T05:51:27.994 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.995 INFO:teuthology.orchestra.run.smithi165.stdout:1764 read 0x250714e thru 0x250f753 (0x8606 bytes) 2024-09-11T05:51:27.995 INFO:teuthology.orchestra.run.smithi165.stdout:1765 read 0x218df82 thru 0x2190fc8 (0x3047 bytes) 2024-09-11T05:51:27.995 INFO:teuthology.orchestra.run.smithi165.stdout:1766 read 0x12a2b6d thru 0x12ab61a (0x8aae bytes) 2024-09-11T05:51:27.995 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:27.995 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.002 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:28.003 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:28.011 INFO:teuthology.orchestra.run.smithi165.stdout:1773 write 0x9943057 thru 0x994d77f (0xa729 bytes) 2024-09-11T05:51:28.014 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:28.010+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:28.020 INFO:teuthology.orchestra.run.smithi165.stdout:1774 punch from 0x3e6010f to 0x3e70036, (0xff27 bytes) 2024-09-11T05:51:28.023 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:28.023 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:28.025 INFO:teuthology.orchestra.run.smithi165.stdout:1775 trunc from 0x994d780 to 0xcc9a3c4 2024-09-11T05:51:28.032 INFO:teuthology.orchestra.run.smithi165.stdout:1776 write 0xa9e6fbd thru 0xa9f680b (0xf84f bytes) 2024-09-11T05:51:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:28.042 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:1777 write 0x9043a8e thru 0x904d76f (0x9ce2 bytes) 2024-09-11T05:51:28.058 INFO:teuthology.orchestra.run.smithi165.stdout:1778 write 0x763c969 thru 0x76426a8 (0x5d40 bytes) 2024-09-11T05:51:28.065 INFO:teuthology.orchestra.run.smithi165.stdout:1779 write 0xb5ee954 thru 0xb5fe7dc (0xfe89 bytes) 2024-09-11T05:51:28.092 INFO:teuthology.orchestra.run.smithi165.stdout:1783 trunc from 0xcc9a3c4 to 0x1d780c5 2024-09-11T05:51:28.103 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.103 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.115 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:28.110+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:28.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:28.148 INFO:teuthology.orchestra.run.smithi165.stdout:1784 trunc from 0x1d780c5 to 0x903ddb9 2024-09-11T05:51:28.156 INFO:teuthology.orchestra.run.smithi165.stdout:1785 write 0x5ce32d3 thru 0x5ceb622 (0x8350 bytes) 2024-09-11T05:51:28.163 INFO:teuthology.orchestra.run.smithi165.stdout:1789 punch from 0x62d68e7 to 0x62e0bfd, (0xa316 bytes) 2024-09-11T05:51:28.163 INFO:teuthology.orchestra.run.smithi165.stdout:1790 punch from 0x64e2cc to 0x64f57d, (0x12b1 bytes) 2024-09-11T05:51:28.172 INFO:teuthology.orchestra.run.smithi165.stdout:1792 write 0xa69c7bc thru 0xa6ac4d1 (0xfd16 bytes) 2024-09-11T05:51:28.180 INFO:teuthology.orchestra.run.smithi165.stdout:1793 write 0x6b8e091 thru 0x6b9bb48 (0xdab8 bytes) 2024-09-11T05:51:28.187 INFO:teuthology.orchestra.run.smithi165.stdout:1796 punch from 0x6df2c4b to 0x6df381e, (0xbd3 bytes) 2024-09-11T05:51:28.187 INFO:teuthology.orchestra.run.smithi165.stdout:1797 punch from 0x8eef90c to 0x8eefe5a, (0x54e bytes) 2024-09-11T05:51:28.187 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.188 INFO:teuthology.orchestra.run.smithi165.stdout:1799 write 0x3f624d5 thru 0x3f6a910 (0x843c bytes) 2024-09-11T05:51:28.193 INFO:teuthology.orchestra.run.smithi165.stdout:1800 read 0x39ef9b3 thru 0x39fbafc (0xc14a bytes) 2024-09-11T05:51:28.195 INFO:teuthology.orchestra.run.smithi165.stdout:1801 trunc from 0xa6ac4d2 to 0xbc57e4f 2024-09-11T05:51:28.202 INFO:teuthology.orchestra.run.smithi165.stdout:1802 write 0xb77bf71 thru 0xb788f73 (0xd003 bytes) 2024-09-11T05:51:28.211 INFO:teuthology.orchestra.run.smithi165.stdout:1804 write 0xb27dd71 thru 0xb27f5ef (0x187f bytes) 2024-09-11T05:51:28.215 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:28.214+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:28.215 INFO:teuthology.orchestra.run.smithi165.stdout:1805 write 0x1b13270 thru 0x1b21b94 (0xe925 bytes) 2024-09-11T05:51:28.227 INFO:teuthology.orchestra.run.smithi165.stdout:1806 read 0x6cba54c thru 0x6cbf80a (0x52bf bytes) 2024-09-11T05:51:28.227 INFO:teuthology.orchestra.run.smithi165.stdout:1808 read 0x299b3a2 thru 0x29a21a5 (0x6e04 bytes) 2024-09-11T05:51:28.227 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.227 INFO:teuthology.orchestra.run.smithi165.stdout:1811 write 0xab017b9 thru 0xab10bc1 (0xf409 bytes) 2024-09-11T05:51:28.235 INFO:teuthology.orchestra.run.smithi165.stdout:1812 write 0x8aff34 thru 0x8bf412 (0xf4df bytes) 2024-09-11T05:51:28.246 INFO:teuthology.orchestra.run.smithi165.stdout:1813 punch from 0x6b6569c to 0x6b689a6, (0x330a bytes) 2024-09-11T05:51:28.246 INFO:teuthology.orchestra.run.smithi165.stdout:1814 write 0x7a5a1ce thru 0x7a614f4 (0x7327 bytes) 2024-09-11T05:51:28.254 INFO:teuthology.orchestra.run.smithi165.stdout:1820 write 0xa99ae7b thru 0xa9a7677 (0xc7fd bytes) 2024-09-11T05:51:28.265 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.266 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.266 INFO:teuthology.orchestra.run.smithi165.stdout:1821 punch from 0xa0aedb6 to 0xa0b7d5a, (0x8fa4 bytes) 2024-09-11T05:51:28.266 INFO:teuthology.orchestra.run.smithi165.stdout:1822 read 0x1e8ac64 thru 0x1e97d24 (0xd0c1 bytes) 2024-09-11T05:51:28.280 INFO:teuthology.orchestra.run.smithi165.stdout:1823 writesame 0xde782aa thru 0xde78ae9 data_size 0x21(0x840 bytes) 2024-09-11T05:51:28.280 INFO:teuthology.orchestra.run.smithi165.stdout:1824 punch from 0xc7bf612 to 0xc7c325f, (0x3c4d bytes) 2024-09-11T05:51:28.280 INFO:teuthology.orchestra.run.smithi165.stdout:1825 punch from 0xb691260 to 0xb696b34, (0x58d4 bytes) 2024-09-11T05:51:28.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:28.287 INFO:teuthology.orchestra.run.smithi165.stdout:1828 write 0xdf345c3 thru 0xdf366f3 (0x2131 bytes) 2024-09-11T05:51:28.291 INFO:teuthology.orchestra.run.smithi165.stdout:1829 read 0xb3f60fe thru 0xb3f776b (0x166e bytes) 2024-09-11T05:51:28.291 INFO:teuthology.orchestra.run.smithi165.stdout:1830 write 0x14d2e66 thru 0x14d7931 (0x4acc bytes) 2024-09-11T05:51:28.308 INFO:teuthology.orchestra.run.smithi165.stdout:1832 trunc from 0xdf366f4 to 0xa7e3b4a 2024-09-11T05:51:28.316 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:28.314+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:28.332 INFO:teuthology.orchestra.run.smithi165.stdout:1833 read 0xa4a5f32 thru 0xa4ad07e (0x714d bytes) 2024-09-11T05:51:28.333 INFO:teuthology.orchestra.run.smithi165.stdout:1835 read 0x49201c0 thru 0x492b082 (0xaec3 bytes) 2024-09-11T05:51:28.333 INFO:teuthology.orchestra.run.smithi165.stdout:1836 write 0xa072c9c thru 0xa080442 (0xd7a7 bytes) 2024-09-11T05:51:28.344 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.344 INFO:teuthology.orchestra.run.smithi165.stdout:1840 punch from 0x91196c7 to 0x9120e29, (0x7762 bytes) 2024-09-11T05:51:28.344 INFO:teuthology.orchestra.run.smithi165.stdout:1842 write 0x8408f2f thru 0x8410919 (0x79eb bytes) 2024-09-11T05:51:28.353 INFO:teuthology.orchestra.run.smithi165.stdout:1843 punch from 0x687f7b3 to 0x6883719, (0x3f66 bytes) 2024-09-11T05:51:28.353 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.353 INFO:teuthology.orchestra.run.smithi165.stdout:1845 read 0x75719b7 thru 0x757f302 (0xd94c bytes) 2024-09-11T05:51:28.353 INFO:teuthology.orchestra.run.smithi165.stdout:1846 read 0x105b774 thru 0x105f69e (0x3f2b bytes) 2024-09-11T05:51:28.354 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.354 INFO:teuthology.orchestra.run.smithi165.stdout:1848 read 0x2d23337 thru 0x2d31935 (0xe5ff bytes) 2024-09-11T05:51:28.354 INFO:teuthology.orchestra.run.smithi165.stdout:1849 punch from 0x50e27ba to 0x50e87d7, (0x601d bytes) 2024-09-11T05:51:28.354 INFO:teuthology.orchestra.run.smithi165.stdout:1850 read 0x36c6029 thru 0x36ce2c8 (0x82a0 bytes) 2024-09-11T05:51:28.355 INFO:teuthology.orchestra.run.smithi165.stdout:1852 read 0x9f6974d thru 0x9f77f20 (0xe7d4 bytes) 2024-09-11T05:51:28.355 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.355 INFO:teuthology.orchestra.run.smithi165.stdout:1854 read 0x9d56436 thru 0x9d62c68 (0xc833 bytes) 2024-09-11T05:51:28.366 INFO:teuthology.orchestra.run.smithi165.stdout:1855 trunc from 0xa7e3b4a to 0xedd82b1 2024-09-11T05:51:28.375 INFO:teuthology.orchestra.run.smithi165.stdout:1858 punch from 0x886b82f to 0x88737b4, (0x7f85 bytes) 2024-09-11T05:51:28.375 INFO:teuthology.orchestra.run.smithi165.stdout:1859 punch from 0x7674d96 to 0x76842ae, (0xf518 bytes) 2024-09-11T05:51:28.375 INFO:teuthology.orchestra.run.smithi165.stdout:1860 punch from 0x857f31c to 0x85816c0, (0x23a4 bytes) 2024-09-11T05:51:28.375 INFO:teuthology.orchestra.run.smithi165.stdout:1861 punch from 0xb5a806a to 0xb5ad741, (0x56d7 bytes) 2024-09-11T05:51:28.375 INFO:teuthology.orchestra.run.smithi165.stdout:1862 write 0xc8fdf71 thru 0xc90a056 (0xc0e6 bytes) 2024-09-11T05:51:28.389 INFO:teuthology.orchestra.run.smithi165.stdout:1864 write 0x2aae1ad thru 0x2ab3eb8 (0x5d0c bytes) 2024-09-11T05:51:28.396 INFO:teuthology.orchestra.run.smithi165.stdout:1866 write 0xc78c6b5 thru 0xc797237 (0xab83 bytes) 2024-09-11T05:51:28.404 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:28.404 INFO:teuthology.orchestra.run.smithi165.stdout:1868 read 0x6b1b77 thru 0x6c0240 (0xe6ca bytes) 2024-09-11T05:51:28.405 INFO:teuthology.orchestra.run.smithi165.stdout:1869 write 0xe84afbb thru 0xe8544f6 (0x953c bytes) 2024-09-11T05:51:28.415 INFO:teuthology.orchestra.run.smithi165.stdout:1870 write 0x3937388 thru 0x393df76 (0x6bef bytes) 2024-09-11T05:51:28.416 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:28.418+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:28.416 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.416 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.424 INFO:teuthology.orchestra.run.smithi165.stdout:1871 writesame 0x252f8ad thru 0x253a7bc data_size 0x8(0xaf10 bytes) 2024-09-11T05:51:28.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:28.439 INFO:teuthology.orchestra.run.smithi165.stdout:1874 read 0xe9b10e8 thru 0xe9bd600 (0xc519 bytes) 2024-09-11T05:51:28.440 INFO:teuthology.orchestra.run.smithi165.stdout:1876 clone 11 order 20 su 131072 sc 3 2024-09-11T05:51:28.517 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:28.514+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:28.595 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.595 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:28.619 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:28.618+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:28.674 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T05:51:28.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:51:28.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T05:51:28.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:28.739 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.739 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.747 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:28.746+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:28.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:28.820 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:28.822+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:28.826 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone10 from 0xedd82b1 (overlap 0x1d780c5) to 0x11ae6dc 2024-09-11T05:51:28.899 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:28.899 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:28.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:28.920 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:28.918+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:29.021 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:29.018+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:29.052 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-09-11T05:51:29.062 INFO:teuthology.orchestra.run.smithi165.stdout:1877 write 0xe2cd0c4 thru 0xe2d692b (0x9868 bytes) 2024-09-11T05:51:29.074 INFO:teuthology.orchestra.run.smithi165.stdout:1879 write 0x252fa8a thru 0x2536e3e (0x73b5 bytes) 2024-09-11T05:51:29.082 INFO:teuthology.orchestra.run.smithi165.stdout:1880 trunc from 0xedd82b1 to 0xeb25c90 2024-09-11T05:51:29.091 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.091 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.096 INFO:teuthology.orchestra.run.smithi165.stdout:1882 read 0x44771ef thru 0x447d7bb (0x65cd bytes) 2024-09-11T05:51:29.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:29.106 INFO:teuthology.orchestra.run.smithi165.stdout:1883 trunc from 0xeb25c90 to 0x8a10244 2024-09-11T05:51:29.113 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:29.113 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:29.122 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:29.118+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:29.127 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.127 INFO:teuthology.orchestra.run.smithi165.stdout:1885 read 0x667a53b thru 0x667b9cb (0x1491 bytes) 2024-09-11T05:51:29.127 INFO:teuthology.orchestra.run.smithi165.stdout:1887 read 0x9e7fcf thru 0x9ef85a (0x788c bytes) 2024-09-11T05:51:29.128 INFO:teuthology.orchestra.run.smithi165.stdout:1889 punch from 0x295696c to 0x2961370, (0xaa04 bytes) 2024-09-11T05:51:29.128 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.128 INFO:teuthology.orchestra.run.smithi165.stdout:1891 read 0x78dbcb1 thru 0x78dff2c (0x427c bytes) 2024-09-11T05:51:29.128 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.128 INFO:teuthology.orchestra.run.smithi165.stdout:1894 write 0x3f3aef4 thru 0x3f3d29a (0x23a7 bytes) 2024-09-11T05:51:29.134 INFO:teuthology.orchestra.run.smithi165.stdout:1895 punch from 0x368b80d to 0x368caba, (0x12ad bytes) 2024-09-11T05:51:29.134 INFO:teuthology.orchestra.run.smithi165.stdout:1896 write 0x29fee8a thru 0x2a018a0 (0x2a17 bytes) 2024-09-11T05:51:29.135 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-11T05:51:28.679425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":211109516,"num_objects":223,"num_object_clones":85,"num_object_copies":446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":51,"num_read":21546,"num_read_kb":78160,"num_write":3630,"num_write_kb":43415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2822,"ondisk_log_size":2822,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287736,"kb_used_data":45352,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754686984,"statfs":{"total":773094113280,"available":772799471616,"internally_reserved":0,"allocated":46440448,"data_stored":73492537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":91951618,"num_objects":60,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":27,"num_read":5394,"num_read_kb":27970,"num_write":489,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051443"},"pg_stats":[{"pgid":"3.1f","version":"37'142","reported_seq":718,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637715+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:51:24.637715+0000","last_peered":"2024-09-11T05:51:24.637715+0000","last_clean":"2024-09-11T05:51:24.637715+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:51:24.637715+0000","last_undegraded":"2024-09-11T05:51:24.637715+0000","last_fullsized":"2024-09-11T05:51:24.637715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:58.117036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":3458944,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":971,"num_read_kb":1687,"num_write":194,"num_write_kb":697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"36'35","reported_seq":322,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.618543+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:51:23.618543+0000","last_peered":"2024-09-11T05:51:23.618543+0000","last_clean":"2024-09-11T05:51:23.618543+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:51:23.618543+0000","last_undegraded":"2024-09-11T05:51:23.618543+0000","last_fullsized":"2024-09-11T05:51:23.618543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:47:09.209965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":3603916,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":238,"num_read_kb":1512,"num_write":33,"num_write_kb":323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"38'37","reported_seq":145,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655073+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:51:24.655073+0000","last_peered":"2024-09-11T05:51:24.655073+0000","last_clean":"2024-09-11T05:51:24.655073+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:51:24.655073+0000","last_undegraded":"2024-09-11T05:51:24.655073+0000","last_fullsized":"2024-09-11T05:51:24.655073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:12.019400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":8603994,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":58,"num_read_kb":3733,"num_write":83,"num_write_kb":2093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"40'36","reported_seq":172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.772424+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:51:24.772424+0000","last_peered":"2024-09-11T05:51:24.772424+0000","last_clean":"2024-09-11T05:51:24.772424+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:51:24.772424+0000","last_undegraded":"2024-09-11T05:51:24.772424+0000","last_fullsized":"2024-09-11T05:51:24.772424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:13.668750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":663552,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":102,"num_read_kb":119,"num_write":33,"num_write_kb":180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"40'57","reported_seq":578,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.776353+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:51:24.776353+0000","last_peered":"2024-09-11T05:51:24.776353+0000","last_clean":"2024-09-11T05:51:24.776353+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:51:24.776353+0000","last_undegraded":"2024-09-11T05:51:24.776353+0000","last_fullsized":"2024-09-11T05:51:24.776353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:50:21.484151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":360448,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":671,"num_read_kb":878,"num_write":55,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"40'84","reported_seq":1131,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.787967+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:51:24.787967+0000","last_peered":"2024-09-11T05:51:24.787967+0000","last_clean":"2024-09-11T05:51:24.787967+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:51:24.787967+0000","last_undegraded":"2024-09-11T05:51:24.787967+0000","last_fullsized":"2024-09-11T05:51:24.787967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:51.992948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":22333145,"num_objects":15,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1135,"num_read_kb":4180,"num_write":116,"num_write_kb":3689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"36'326","reported_seq":737,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631070+0000","last_change":"2024-09-11T05:51:07.514037+0000","last_active":"2024-09-11T05:51:24.631070+0000","last_peered":"2024-09-11T05:51:24.631070+0000","last_clean":"2024-09-11T05:51:24.631070+0000","last_became_active":"2024-09-11T05:51:05.507475+0000","last_became_peered":"2024-09-11T05:51:05.507475+0000","last_unstale":"2024-09-11T05:51:24.631070+0000","last_undegraded":"2024-09-11T05:51:24.631070+0000","last_fullsized":"2024-09-11T05:51:24.631070+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:42.027569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":785266,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":2323,"num_read_kb":2459,"num_write":509,"num_write_kb":733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"38'85","reported_seq":674,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.991564+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:51:23.991564+0000","last_peered":"2024-09-11T05:51:23.991564+0000","last_clean":"2024-09-11T05:51:23.991564+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:51:23.991564+0000","last_undegraded":"2024-09-11T05:51:23.991564+0000","last_fullsized":"2024-09-11T05:51:23.991564+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:07:00.747866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":4731870,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1062,"num_read_kb":2814,"num_write":82,"num_write_kb":1325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"39'67","reported_seq":351,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637893+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:51:24.637893+0000","last_peered":"2024-09-11T05:51:24.637893+0000","last_clean":"2024-09-11T05:51:24.637893+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:51:24.637893+0000","last_undegraded":"2024-09-11T05:51:24.637893+0000","last_fullsized":"2024-09-11T05:51:24.637893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:46:39.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":131072,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":673,"num_read_kb":566,"num_write":81,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"36'26","reported_seq":321,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655427+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:51:24.655427+0000","last_peered":"2024-09-11T05:51:24.655427+0000","last_clean":"2024-09-11T05:51:24.655427+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:51:24.655427+0000","last_undegraded":"2024-09-11T05:51:24.655427+0000","last_fullsized":"2024-09-11T05:51:24.655427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:34:44.021283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":7548928,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":243,"num_read_kb":1536,"num_write":24,"num_write_kb":246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'32","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683658+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:51:24.683658+0000","last_peered":"2024-09-11T05:51:24.683658+0000","last_clean":"2024-09-11T05:51:24.683658+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:51:24.683658+0000","last_undegraded":"2024-09-11T05:51:24.683658+0000","last_fullsized":"2024-09-11T05:51:24.683658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:32:32.943776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":17335406,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":11,"num_read_kb":47,"num_write":32,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"38'187","reported_seq":1637,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631764+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:51:24.631764+0000","last_peered":"2024-09-11T05:51:24.631764+0000","last_clean":"2024-09-11T05:51:24.631764+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:51:24.631764+0000","last_undegraded":"2024-09-11T05:51:24.631764+0000","last_fullsized":"2024-09-11T05:51:24.631764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:04.745632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":9823498,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":2632,"num_read_kb":7808,"num_write":235,"num_write_kb":4464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683904+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:51:24.683904+0000","last_peered":"2024-09-11T05:51:24.683904+0000","last_clean":"2024-09-11T05:51:24.683904+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:51:24.683904+0000","last_undegraded":"2024-09-11T05:51:24.683904+0000","last_fullsized":"2024-09-11T05:51:24.683904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"38'35","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637973+0000","last_change":"2024-09-11T05:51:07.518766+0000","last_active":"2024-09-11T05:51:24.637973+0000","last_peered":"2024-09-11T05:51:24.637973+0000","last_clean":"2024-09-11T05:51:24.637973+0000","last_became_active":"2024-09-11T05:51:05.499998+0000","last_became_peered":"2024-09-11T05:51:05.499998+0000","last_unstale":"2024-09-11T05:51:24.637973+0000","last_undegraded":"2024-09-11T05:51:24.637973+0000","last_fullsized":"2024-09-11T05:51:24.637973+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:57:04.214848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":5521408,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":1474,"num_write":45,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":55,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.633140+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:51:24.633140+0000","last_peered":"2024-09-11T05:51:24.633140+0000","last_clean":"2024-09-11T05:51:24.633140+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:51:24.633140+0000","last_undegraded":"2024-09-11T05:51:24.633140+0000","last_fullsized":"2024-09-11T05:51:24.633140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"39'218","reported_seq":291,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.924816+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:51:23.924816+0000","last_peered":"2024-09-11T05:51:23.924816+0000","last_clean":"2024-09-11T05:51:23.924816+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:51:23.924816+0000","last_undegraded":"2024-09-11T05:51:23.924816+0000","last_fullsized":"2024-09-11T05:51:23.924816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:18:50.534620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":87,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":777,"num_write":343,"num_write_kb":596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683936+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:51:24.683936+0000","last_peered":"2024-09-11T05:51:24.683936+0000","last_clean":"2024-09-11T05:51:24.683936+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:51:24.683936+0000","last_undegraded":"2024-09-11T05:51:24.683936+0000","last_fullsized":"2024-09-11T05:51:24.683936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"35'55","reported_seq":488,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.632450+0000","last_change":"2024-09-11T05:51:07.514746+0000","last_active":"2024-09-11T05:51:24.632450+0000","last_peered":"2024-09-11T05:51:24.632450+0000","last_clean":"2024-09-11T05:51:24.632450+0000","last_became_active":"2024-09-11T05:51:06.470494+0000","last_became_peered":"2024-09-11T05:51:06.470494+0000","last_unstale":"2024-09-11T05:51:24.632450+0000","last_undegraded":"2024-09-11T05:51:24.632450+0000","last_fullsized":"2024-09-11T05:51:24.632450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:29.127159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":6864896,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":399,"num_read_kb":5045,"num_write":55,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.633374+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:51:24.633374+0000","last_peered":"2024-09-11T05:51:24.633374+0000","last_clean":"2024-09-11T05:51:24.633374+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:51:24.633374+0000","last_undegraded":"2024-09-11T05:51:24.633374+0000","last_fullsized":"2024-09-11T05:51:24.633374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'112","reported_seq":848,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.796032+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:51:24.796032+0000","last_peered":"2024-09-11T05:51:24.796032+0000","last_clean":"2024-09-11T05:51:24.796032+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:51:24.796032+0000","last_undegraded":"2024-09-11T05:51:24.796032+0000","last_fullsized":"2024-09-11T05:51:24.796032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:04:57.991416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":4258914,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1914,"num_read_kb":2222,"num_write":104,"num_write_kb":222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"40'59","reported_seq":776,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.678267+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:51:24.678267+0000","last_peered":"2024-09-11T05:51:24.678267+0000","last_clean":"2024-09-11T05:51:24.678267+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:51:24.678267+0000","last_undegraded":"2024-09-11T05:51:24.678267+0000","last_fullsized":"2024-09-11T05:51:24.678267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:39.490356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":3535266,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":688,"num_read_kb":2864,"num_write":58,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.654951+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:51:24.654951+0000","last_peered":"2024-09-11T05:51:24.654951+0000","last_clean":"2024-09-11T05:51:24.654951+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:51:24.654951+0000","last_undegraded":"2024-09-11T05:51:24.654951+0000","last_fullsized":"2024-09-11T05:51:24.654951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"40'58","reported_seq":382,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.736409+0000","last_change":"2024-09-11T05:51:07.514375+0000","last_active":"2024-09-11T05:51:24.736409+0000","last_peered":"2024-09-11T05:51:24.736409+0000","last_clean":"2024-09-11T05:51:24.736409+0000","last_became_active":"2024-09-11T05:51:06.471743+0000","last_became_peered":"2024-09-11T05:51:06.471743+0000","last_unstale":"2024-09-11T05:51:24.736409+0000","last_undegraded":"2024-09-11T05:51:24.736409+0000","last_fullsized":"2024-09-11T05:51:24.736409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:00.985014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":12295444,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":294,"num_read_kb":1767,"num_write":53,"num_write_kb":1619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.733560+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:51:24.733560+0000","last_peered":"2024-09-11T05:51:24.733560+0000","last_clean":"2024-09-11T05:51:24.733560+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:51:24.733560+0000","last_undegraded":"2024-09-11T05:51:24.733560+0000","last_fullsized":"2024-09-11T05:51:24.733560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"39'76","reported_seq":379,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.676741+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:51:24.676741+0000","last_peered":"2024-09-11T05:51:24.676741+0000","last_clean":"2024-09-11T05:51:24.676741+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:51:24.676741+0000","last_undegraded":"2024-09-11T05:51:24.676741+0000","last_fullsized":"2024-09-11T05:51:24.676741+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:01:14.318081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":4063280,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":729,"num_read_kb":4404,"num_write":118,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683518+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:51:24.683518+0000","last_peered":"2024-09-11T05:51:24.683518+0000","last_clean":"2024-09-11T05:51:24.683518+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:51:24.683518+0000","last_undegraded":"2024-09-11T05:51:24.683518+0000","last_fullsized":"2024-09-11T05:51:24.683518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"36'120","reported_seq":560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.618948+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:51:23.618948+0000","last_peered":"2024-09-11T05:51:23.618948+0000","last_clean":"2024-09-11T05:51:23.618948+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:51:23.618948+0000","last_undegraded":"2024-09-11T05:51:23.618948+0000","last_fullsized":"2024-09-11T05:51:23.618948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:55:55.029570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":3999862,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":922,"num_read_kb":5167,"num_write":191,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.654982+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:51:24.654982+0000","last_peered":"2024-09-11T05:51:24.654982+0000","last_clean":"2024-09-11T05:51:24.654982+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:51:24.654982+0000","last_undegraded":"2024-09-11T05:51:24.654982+0000","last_fullsized":"2024-09-11T05:51:24.654982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":205,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683577+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:51:24.683577+0000","last_peered":"2024-09-11T05:51:24.683577+0000","last_clean":"2024-09-11T05:51:24.683577+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:51:24.683577+0000","last_undegraded":"2024-09-11T05:51:24.683577+0000","last_fullsized":"2024-09-11T05:51:24.683577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"38'42","reported_seq":399,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.718280+0000","last_change":"2024-09-11T05:51:07.507117+0000","last_active":"2024-09-11T05:51:23.718280+0000","last_peered":"2024-09-11T05:51:23.718280+0000","last_clean":"2024-09-11T05:51:23.718280+0000","last_became_active":"2024-09-11T05:51:05.508345+0000","last_became_peered":"2024-09-11T05:51:05.508345+0000","last_unstale":"2024-09-11T05:51:23.718280+0000","last_undegraded":"2024-09-11T05:51:23.718280+0000","last_fullsized":"2024-09-11T05:51:23.718280+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:10:16.718724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":5442445,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":324,"num_read_kb":1939,"num_write":51,"num_write_kb":1395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"40'47","reported_seq":521,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631163+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:51:24.631163+0000","last_peered":"2024-09-11T05:51:24.631163+0000","last_clean":"2024-09-11T05:51:24.631163+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:51:24.631163+0000","last_undegraded":"2024-09-11T05:51:24.631163+0000","last_fullsized":"2024-09-11T05:51:24.631163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:59.972089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":1899707,"num_objects":10,"num_object_clones":5,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":4,"num_read":448,"num_read_kb":738,"num_write":40,"num_write_kb":242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'111","reported_seq":697,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.803892+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:51:24.803892+0000","last_peered":"2024-09-11T05:51:24.803892+0000","last_clean":"2024-09-11T05:51:24.803892+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:51:24.803892+0000","last_undegraded":"2024-09-11T05:51:24.803892+0000","last_fullsized":"2024-09-11T05:51:24.803892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:08:04.607370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2387013,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":1387,"num_read_kb":2646,"num_write":119,"num_write_kb":1277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"39'73","reported_seq":440,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.932779+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:51:23.932779+0000","last_peered":"2024-09-11T05:51:23.932779+0000","last_clean":"2024-09-11T05:51:23.932779+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:51:23.932779+0000","last_undegraded":"2024-09-11T05:51:23.932779+0000","last_fullsized":"2024-09-11T05:51:23.932779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:50.343900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":8810586,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":340,"num_read_kb":4431,"num_write":93,"num_write_kb":3827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"35'12","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655112+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:51:24.655112+0000","last_peered":"2024-09-11T05:51:24.655112+0000","last_clean":"2024-09-11T05:51:24.655112+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:51:24.655112+0000","last_undegraded":"2024-09-11T05:51:24.655112+0000","last_fullsized":"2024-09-11T05:51:24.655112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:18.730528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"38'51","reported_seq":609,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683811+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:51:24.683811+0000","last_peered":"2024-09-11T05:51:24.683811+0000","last_clean":"2024-09-11T05:51:24.683811+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:51:24.683811+0000","last_undegraded":"2024-09-11T05:51:24.683811+0000","last_fullsized":"2024-09-11T05:51:24.683811+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:38:34.205608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":7815960,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":529,"num_read_kb":6088,"num_write":52,"num_write_kb":2131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"40'82","reported_seq":623,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.735520+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:51:24.735520+0000","last_peered":"2024-09-11T05:51:24.735520+0000","last_clean":"2024-09-11T05:51:24.735520+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:51:24.735520+0000","last_undegraded":"2024-09-11T05:51:24.735520+0000","last_fullsized":"2024-09-11T05:51:24.735520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:17:34.042628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":28082230,"num_objects":11,"num_object_clones":5,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":530,"num_read_kb":1265,"num_write":78,"num_write_kb":1194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"40'80","reported_seq":159,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.763157+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:51:24.763157+0000","last_peered":"2024-09-11T05:51:24.763157+0000","last_clean":"2024-09-11T05:51:24.763157+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:51:24.763157+0000","last_undegraded":"2024-09-11T05:51:24.763157+0000","last_fullsized":"2024-09-11T05:51:24.763157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:18.066446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":12539208,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":353,"num_read_kb":7051,"num_write":120,"num_write_kb":3657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"40'51","reported_seq":293,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.638198+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:51:24.638198+0000","last_peered":"2024-09-11T05:51:24.638198+0000","last_clean":"2024-09-11T05:51:24.638198+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:51:24.638198+0000","last_undegraded":"2024-09-11T05:51:24.638198+0000","last_fullsized":"2024-09-11T05:51:24.638198+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:33.440079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9883022,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":220,"num_read_kb":840,"num_write":45,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"38'57","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683535+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:51:24.683535+0000","last_peered":"2024-09-11T05:51:24.683535+0000","last_clean":"2024-09-11T05:51:24.683535+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:51:24.683535+0000","last_undegraded":"2024-09-11T05:51:24.683535+0000","last_fullsized":"2024-09-11T05:51:24.683535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:35.362300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":8696030,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":25,"num_read_kb":179,"num_write":54,"num_write_kb":502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"40'209","reported_seq":510,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.638144+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:51:24.638144+0000","last_peered":"2024-09-11T05:51:24.638144+0000","last_clean":"2024-09-11T05:51:24.638144+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:51:24.638144+0000","last_undegraded":"2024-09-11T05:51:24.638144+0000","last_fullsized":"2024-09-11T05:51:24.638144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:28:59.732626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":3215458,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":1138,"num_read_kb":1824,"num_write":322,"num_write_kb":499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"40'40","reported_seq":92,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683752+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:51:24.683752+0000","last_peered":"2024-09-11T05:51:24.683752+0000","last_clean":"2024-09-11T05:51:24.683752+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:51:24.683752+0000","last_undegraded":"2024-09-11T05:51:24.683752+0000","last_fullsized":"2024-09-11T05:51:24.683752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:08.002000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":1828274,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":3,"num_read_kb":0,"num_write":39,"num_write_kb":496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":210519129,"num_objects":219,"num_object_clones":85,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":51,"num_read":21430,"num_read_kb":78060,"num_write":3469,"num_write_kb":40997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42029056,"data_stored":71084846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2702,"ondisk_log_size":2702,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":9712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":9945088,"data_stored":15114952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1572864,"data_stored":2294417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36228,"kb_used_data":3832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335612,"statfs":{"total":96636764160,"available":96599666688,"internally_reserved":0,"allocated":3923968,"data_stored":6461524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44820,"kb_used_data":13636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327020,"statfs":{"total":96636764160,"available":96590868480,"internally_reserved":0,"allocated":13963264,"data_stored":23681207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34196,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337644,"statfs":{"total":96636764160,"available":96601747456,"internally_reserved":0,"allocated":3084288,"data_stored":4641456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29168,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342672,"statfs":{"total":96636764160,"available":96606896128,"internally_reserved":0,"allocated":2187264,"data_stored":2827440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":5197824,"data_stored":8100318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37596,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334244,"statfs":{"total":96636764160,"available":96598265856,"internally_reserved":0,"allocated":6565888,"data_stored":10371223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5545984,"data_stored":9601304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4759552,"data_stored":7920651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1753088,"data_stored":2647800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2650112,"data_stored":4461816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13635584,"data_stored":23588195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3485696,"data_stored":6281865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1273856,"data_stored":2238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8925184,"data_stored":14345011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:29.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:29.146 INFO:teuthology.orchestra.run.smithi165.stdout:1897 trunc from 0x8a10244 to 0xd24e27f 2024-09-11T05:51:29.152 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.152 INFO:teuthology.orchestra.run.smithi165.stdout:1900 punch from 0x79bb10c to 0x79bee2c, (0x3d20 bytes) 2024-09-11T05:51:29.152 INFO:teuthology.orchestra.run.smithi165.stdout:1901 write 0xcfde8f7 thru 0xcfea664 (0xbd6e bytes) 2024-09-11T05:51:29.160 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.160 INFO:teuthology.orchestra.run.smithi165.stdout:1904 writesame 0x92c5337 thru 0x92cafba data_size 0x3e(0x5c84 bytes) 2024-09-11T05:51:29.164 INFO:teuthology.orchestra.run.smithi165.stdout:1905 read 0x25cf4ca thru 0x25d7077 (0x7bae bytes) 2024-09-11T05:51:29.179 INFO:teuthology.orchestra.run.smithi165.stdout:1906 trunc from 0xd24e27f to 0x316ac5c 2024-09-11T05:51:29.201 INFO:teuthology.orchestra.run.smithi165.stdout:1907 trunc from 0x316ac5c to 0x31a8e16 2024-09-11T05:51:29.219 INFO:teuthology.orchestra.run.smithi165.stdout:1908 trunc from 0x31a8e16 to 0xa71d0b3 2024-09-11T05:51:29.223 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:29.222+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:29.233 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.233 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.240 INFO:teuthology.orchestra.run.smithi165.stdout:1909 write 0xe5cac42 thru 0xe5d37f5 (0x8bb4 bytes) 2024-09-11T05:51:29.245 INFO:teuthology.orchestra.run.smithi165.stdout:1910 punch from 0x8038624 to 0x803f823, (0x71ff bytes) 2024-09-11T05:51:29.245 INFO:teuthology.orchestra.run.smithi165.stdout:1911 writesame 0xc87dbc4 thru 0xc88b89c data_size 0x1(0xdcd9 bytes) 2024-09-11T05:51:29.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:29.292 INFO:teuthology.orchestra.run.smithi165.stdout:1912 trunc from 0xe5d37f6 to 0x3358c37 2024-09-11T05:51:29.311 INFO:teuthology.orchestra.run.smithi165.stdout:1914 trunc from 0x3358c37 to 0x27a850c 2024-09-11T05:51:29.324 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:29.326+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:29.342 INFO:teuthology.orchestra.run.smithi165.stdout:1915 trunc from 0x27a850c to 0xdbe8059 2024-09-11T05:51:29.349 INFO:teuthology.orchestra.run.smithi165.stdout:1916 write 0xa9f9351 thru 0xa9fcd2b (0x39db bytes) 2024-09-11T05:51:29.353 INFO:teuthology.orchestra.run.smithi165.stdout:1917 write 0xcac8a31 thru 0xcad2937 (0x9f07 bytes) 2024-09-11T05:51:29.357 INFO:teuthology.orchestra.run.smithi165.stdout:1918 writesame 0x91987e2 thru 0x919fd03 data_size 0x3a(0x7522 bytes) 2024-09-11T05:51:29.361 INFO:teuthology.orchestra.run.smithi165.stdout:1919 writesame 0xafdaf94 thru 0xafe46a0 data_size 0x1(0x970d bytes) 2024-09-11T05:51:29.374 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.375 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.387 INFO:teuthology.orchestra.run.smithi165.stdout:1920 writesame 0xe5facb7 thru 0xe602bfd data_size 0x1(0x7f47 bytes) 2024-09-11T05:51:29.387 INFO:teuthology.orchestra.run.smithi165.stdout:1921 write 0x701689a thru 0x702025d (0x99c4 bytes) 2024-09-11T05:51:29.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:29.391 INFO:teuthology.orchestra.run.smithi165.stdout:1922 write 0xbab2a6d thru 0xbabc3f5 (0x9989 bytes) 2024-09-11T05:51:29.394 INFO:teuthology.orchestra.run.smithi165.stdout:1924 read 0x176d38f thru 0x1774637 (0x72a9 bytes) 2024-09-11T05:51:29.395 INFO:teuthology.orchestra.run.smithi165.stdout:1925 trunc from 0xe602bfe to 0xee005c1 2024-09-11T05:51:29.403 INFO:teuthology.orchestra.run.smithi165.stdout:1926 read 0x10f197c thru 0x10fe659 (0xccde bytes) 2024-09-11T05:51:29.404 INFO:teuthology.orchestra.run.smithi165.stdout:1928 read 0x34c481f thru 0x34c78a3 (0x3085 bytes) 2024-09-11T05:51:29.404 INFO:teuthology.orchestra.run.smithi165.stdout:1929 write 0x36f8e84 thru 0x3700e1b (0x7f98 bytes) 2024-09-11T05:51:29.408 INFO:teuthology.orchestra.run.smithi165.stdout:1930 write 0xceeb966 thru 0xcef5d02 (0xa39d bytes) 2024-09-11T05:51:29.422 INFO:teuthology.orchestra.run.smithi165.stdout:1931 trunc from 0xee005c1 to 0x746ea4f 2024-09-11T05:51:29.425 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:29.422+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:29.440 INFO:teuthology.orchestra.run.smithi165.stdout:1932 trunc from 0x746ea4f to 0x2f5ac7b 2024-09-11T05:51:29.451 INFO:teuthology.orchestra.run.smithi165.stdout:1933 read 0xc0ebd2 thru 0xc13aef (0x4f1e bytes) 2024-09-11T05:51:29.452 INFO:teuthology.orchestra.run.smithi165.stdout:1934 trunc from 0x2f5ac7b to 0x269a9be 2024-09-11T05:51:29.483 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:29.484 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:29.484 INFO:teuthology.orchestra.run.smithi165.stdout:1935 trunc from 0x269a9be to 0x776a830 2024-09-11T05:51:29.493 INFO:teuthology.orchestra.run.smithi165.stdout:1936 punch from 0x63cc673 to 0x63d825a, (0xbbe7 bytes) 2024-09-11T05:51:29.493 INFO:teuthology.orchestra.run.smithi165.stdout:1937 read 0x3ea5b83 thru 0x3eb0863 (0xace1 bytes) 2024-09-11T05:51:29.493 INFO:teuthology.orchestra.run.smithi165.stdout:1938 write 0x4ea78a5 thru 0x4eaa910 (0x306c bytes) 2024-09-11T05:51:29.499 INFO:teuthology.orchestra.run.smithi165.stdout:1939 read 0x17e6f89 thru 0x17ec646 (0x56be bytes) 2024-09-11T05:51:29.499 INFO:teuthology.orchestra.run.smithi165.stdout:1942 punch from 0x181aa4f to 0x181d533, (0x2ae4 bytes) 2024-09-11T05:51:29.499 INFO:teuthology.orchestra.run.smithi165.stdout:1943 read 0xfe06c5 thru 0xfed706 (0xd042 bytes) 2024-09-11T05:51:29.499 INFO:teuthology.orchestra.run.smithi165.stdout:1944 read 0x6f4d15a thru 0x6f55a1e (0x88c5 bytes) 2024-09-11T05:51:29.499 INFO:teuthology.orchestra.run.smithi165.stdout:1948 write 0x6953dd6 thru 0x695d087 (0x92b2 bytes) 2024-09-11T05:51:29.503 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-11T05:51:28.679425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":211109516,"num_objects":223,"num_object_clones":85,"num_object_copies":446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":51,"num_read":21546,"num_read_kb":78160,"num_write":3630,"num_write_kb":43415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2822,"ondisk_log_size":2822,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287736,"kb_used_data":45352,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754686984,"statfs":{"total":773094113280,"available":772799471616,"internally_reserved":0,"allocated":46440448,"data_stored":73492537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":91951618,"num_objects":60,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":27,"num_read":5394,"num_read_kb":27970,"num_write":489,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051443"},"pg_stats":[{"pgid":"3.1f","version":"37'142","reported_seq":718,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637715+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:51:24.637715+0000","last_peered":"2024-09-11T05:51:24.637715+0000","last_clean":"2024-09-11T05:51:24.637715+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:51:24.637715+0000","last_undegraded":"2024-09-11T05:51:24.637715+0000","last_fullsized":"2024-09-11T05:51:24.637715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:51:58.117036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":3458944,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":971,"num_read_kb":1687,"num_write":194,"num_write_kb":697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"36'35","reported_seq":322,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.618543+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:51:23.618543+0000","last_peered":"2024-09-11T05:51:23.618543+0000","last_clean":"2024-09-11T05:51:23.618543+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:51:23.618543+0000","last_undegraded":"2024-09-11T05:51:23.618543+0000","last_fullsized":"2024-09-11T05:51:23.618543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:47:09.209965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":3603916,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":238,"num_read_kb":1512,"num_write":33,"num_write_kb":323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"38'37","reported_seq":145,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655073+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:51:24.655073+0000","last_peered":"2024-09-11T05:51:24.655073+0000","last_clean":"2024-09-11T05:51:24.655073+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:51:24.655073+0000","last_undegraded":"2024-09-11T05:51:24.655073+0000","last_fullsized":"2024-09-11T05:51:24.655073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:12.019400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":8603994,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":58,"num_read_kb":3733,"num_write":83,"num_write_kb":2093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"40'36","reported_seq":172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.772424+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:51:24.772424+0000","last_peered":"2024-09-11T05:51:24.772424+0000","last_clean":"2024-09-11T05:51:24.772424+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:51:24.772424+0000","last_undegraded":"2024-09-11T05:51:24.772424+0000","last_fullsized":"2024-09-11T05:51:24.772424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:13.668750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":663552,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":102,"num_read_kb":119,"num_write":33,"num_write_kb":180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"40'57","reported_seq":578,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.776353+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:51:24.776353+0000","last_peered":"2024-09-11T05:51:24.776353+0000","last_clean":"2024-09-11T05:51:24.776353+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:51:24.776353+0000","last_undegraded":"2024-09-11T05:51:24.776353+0000","last_fullsized":"2024-09-11T05:51:24.776353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:50:21.484151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":360448,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":671,"num_read_kb":878,"num_write":55,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"40'84","reported_seq":1131,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.787967+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:51:24.787967+0000","last_peered":"2024-09-11T05:51:24.787967+0000","last_clean":"2024-09-11T05:51:24.787967+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:51:24.787967+0000","last_undegraded":"2024-09-11T05:51:24.787967+0000","last_fullsized":"2024-09-11T05:51:24.787967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:51.992948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":22333145,"num_objects":15,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1135,"num_read_kb":4180,"num_write":116,"num_write_kb":3689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"36'326","reported_seq":737,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631070+0000","last_change":"2024-09-11T05:51:07.514037+0000","last_active":"2024-09-11T05:51:24.631070+0000","last_peered":"2024-09-11T05:51:24.631070+0000","last_clean":"2024-09-11T05:51:24.631070+0000","last_became_active":"2024-09-11T05:51:05.507475+0000","last_became_peered":"2024-09-11T05:51:05.507475+0000","last_unstale":"2024-09-11T05:51:24.631070+0000","last_undegraded":"2024-09-11T05:51:24.631070+0000","last_fullsized":"2024-09-11T05:51:24.631070+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:42.027569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":785266,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":2323,"num_read_kb":2459,"num_write":509,"num_write_kb":733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"38'85","reported_seq":674,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.991564+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:51:23.991564+0000","last_peered":"2024-09-11T05:51:23.991564+0000","last_clean":"2024-09-11T05:51:23.991564+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:51:23.991564+0000","last_undegraded":"2024-09-11T05:51:23.991564+0000","last_fullsized":"2024-09-11T05:51:23.991564+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:07:00.747866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":4731870,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1062,"num_read_kb":2814,"num_write":82,"num_write_kb":1325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"39'67","reported_seq":351,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637893+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:51:24.637893+0000","last_peered":"2024-09-11T05:51:24.637893+0000","last_clean":"2024-09-11T05:51:24.637893+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:51:24.637893+0000","last_undegraded":"2024-09-11T05:51:24.637893+0000","last_fullsized":"2024-09-11T05:51:24.637893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:46:39.383795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":131072,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":673,"num_read_kb":566,"num_write":81,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"36'26","reported_seq":321,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655427+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:51:24.655427+0000","last_peered":"2024-09-11T05:51:24.655427+0000","last_clean":"2024-09-11T05:51:24.655427+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:51:24.655427+0000","last_undegraded":"2024-09-11T05:51:24.655427+0000","last_fullsized":"2024-09-11T05:51:24.655427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:34:44.021283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":7548928,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":243,"num_read_kb":1536,"num_write":24,"num_write_kb":246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'32","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683658+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:51:24.683658+0000","last_peered":"2024-09-11T05:51:24.683658+0000","last_clean":"2024-09-11T05:51:24.683658+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:51:24.683658+0000","last_undegraded":"2024-09-11T05:51:24.683658+0000","last_fullsized":"2024-09-11T05:51:24.683658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:32:32.943776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":17335406,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":11,"num_read_kb":47,"num_write":32,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"38'187","reported_seq":1637,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631764+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:51:24.631764+0000","last_peered":"2024-09-11T05:51:24.631764+0000","last_clean":"2024-09-11T05:51:24.631764+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:51:24.631764+0000","last_undegraded":"2024-09-11T05:51:24.631764+0000","last_fullsized":"2024-09-11T05:51:24.631764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:04.745632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":9823498,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":2632,"num_read_kb":7808,"num_write":235,"num_write_kb":4464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683904+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:51:24.683904+0000","last_peered":"2024-09-11T05:51:24.683904+0000","last_clean":"2024-09-11T05:51:24.683904+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:51:24.683904+0000","last_undegraded":"2024-09-11T05:51:24.683904+0000","last_fullsized":"2024-09-11T05:51:24.683904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"38'35","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.637973+0000","last_change":"2024-09-11T05:51:07.518766+0000","last_active":"2024-09-11T05:51:24.637973+0000","last_peered":"2024-09-11T05:51:24.637973+0000","last_clean":"2024-09-11T05:51:24.637973+0000","last_became_active":"2024-09-11T05:51:05.499998+0000","last_became_peered":"2024-09-11T05:51:05.499998+0000","last_unstale":"2024-09-11T05:51:24.637973+0000","last_undegraded":"2024-09-11T05:51:24.637973+0000","last_fullsized":"2024-09-11T05:51:24.637973+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:57:04.214848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":5521408,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":1474,"num_write":45,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":55,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.633140+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:51:24.633140+0000","last_peered":"2024-09-11T05:51:24.633140+0000","last_clean":"2024-09-11T05:51:24.633140+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:51:24.633140+0000","last_undegraded":"2024-09-11T05:51:24.633140+0000","last_fullsized":"2024-09-11T05:51:24.633140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"39'218","reported_seq":291,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.924816+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:51:23.924816+0000","last_peered":"2024-09-11T05:51:23.924816+0000","last_clean":"2024-09-11T05:51:23.924816+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:51:23.924816+0000","last_undegraded":"2024-09-11T05:51:23.924816+0000","last_fullsized":"2024-09-11T05:51:23.924816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:18:50.534620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":87,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":777,"num_write":343,"num_write_kb":596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683936+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:51:24.683936+0000","last_peered":"2024-09-11T05:51:24.683936+0000","last_clean":"2024-09-11T05:51:24.683936+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:51:24.683936+0000","last_undegraded":"2024-09-11T05:51:24.683936+0000","last_fullsized":"2024-09-11T05:51:24.683936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"35'55","reported_seq":488,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.632450+0000","last_change":"2024-09-11T05:51:07.514746+0000","last_active":"2024-09-11T05:51:24.632450+0000","last_peered":"2024-09-11T05:51:24.632450+0000","last_clean":"2024-09-11T05:51:24.632450+0000","last_became_active":"2024-09-11T05:51:06.470494+0000","last_became_peered":"2024-09-11T05:51:06.470494+0000","last_unstale":"2024-09-11T05:51:24.632450+0000","last_undegraded":"2024-09-11T05:51:24.632450+0000","last_fullsized":"2024-09-11T05:51:24.632450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:29.127159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":6864896,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":399,"num_read_kb":5045,"num_write":55,"num_write_kb":1064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.633374+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:51:24.633374+0000","last_peered":"2024-09-11T05:51:24.633374+0000","last_clean":"2024-09-11T05:51:24.633374+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:51:24.633374+0000","last_undegraded":"2024-09-11T05:51:24.633374+0000","last_fullsized":"2024-09-11T05:51:24.633374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"38'112","reported_seq":848,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.796032+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:51:24.796032+0000","last_peered":"2024-09-11T05:51:24.796032+0000","last_clean":"2024-09-11T05:51:24.796032+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:51:24.796032+0000","last_undegraded":"2024-09-11T05:51:24.796032+0000","last_fullsized":"2024-09-11T05:51:24.796032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:04:57.991416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":4258914,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1914,"num_read_kb":2222,"num_write":104,"num_write_kb":222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"40'59","reported_seq":776,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.678267+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:51:24.678267+0000","last_peered":"2024-09-11T05:51:24.678267+0000","last_clean":"2024-09-11T05:51:24.678267+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:51:24.678267+0000","last_undegraded":"2024-09-11T05:51:24.678267+0000","last_fullsized":"2024-09-11T05:51:24.678267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:39.490356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":3535266,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":688,"num_read_kb":2864,"num_write":58,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.654951+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:51:24.654951+0000","last_peered":"2024-09-11T05:51:24.654951+0000","last_clean":"2024-09-11T05:51:24.654951+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:51:24.654951+0000","last_undegraded":"2024-09-11T05:51:24.654951+0000","last_fullsized":"2024-09-11T05:51:24.654951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"40'58","reported_seq":382,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.736409+0000","last_change":"2024-09-11T05:51:07.514375+0000","last_active":"2024-09-11T05:51:24.736409+0000","last_peered":"2024-09-11T05:51:24.736409+0000","last_clean":"2024-09-11T05:51:24.736409+0000","last_became_active":"2024-09-11T05:51:06.471743+0000","last_became_peered":"2024-09-11T05:51:06.471743+0000","last_unstale":"2024-09-11T05:51:24.736409+0000","last_undegraded":"2024-09-11T05:51:24.736409+0000","last_fullsized":"2024-09-11T05:51:24.736409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:00.985014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":12295444,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":294,"num_read_kb":1767,"num_write":53,"num_write_kb":1619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.733560+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:51:24.733560+0000","last_peered":"2024-09-11T05:51:24.733560+0000","last_clean":"2024-09-11T05:51:24.733560+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:51:24.733560+0000","last_undegraded":"2024-09-11T05:51:24.733560+0000","last_fullsized":"2024-09-11T05:51:24.733560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"39'76","reported_seq":379,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.676741+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:51:24.676741+0000","last_peered":"2024-09-11T05:51:24.676741+0000","last_clean":"2024-09-11T05:51:24.676741+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:51:24.676741+0000","last_undegraded":"2024-09-11T05:51:24.676741+0000","last_fullsized":"2024-09-11T05:51:24.676741+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:01:14.318081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":4063280,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":729,"num_read_kb":4404,"num_write":118,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683518+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:51:24.683518+0000","last_peered":"2024-09-11T05:51:24.683518+0000","last_clean":"2024-09-11T05:51:24.683518+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:51:24.683518+0000","last_undegraded":"2024-09-11T05:51:24.683518+0000","last_fullsized":"2024-09-11T05:51:24.683518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"36'120","reported_seq":560,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.618948+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:51:23.618948+0000","last_peered":"2024-09-11T05:51:23.618948+0000","last_clean":"2024-09-11T05:51:23.618948+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:51:23.618948+0000","last_undegraded":"2024-09-11T05:51:23.618948+0000","last_fullsized":"2024-09-11T05:51:23.618948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:55:55.029570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":3999862,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":922,"num_read_kb":5167,"num_write":191,"num_write_kb":217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":56,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.654982+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:51:24.654982+0000","last_peered":"2024-09-11T05:51:24.654982+0000","last_clean":"2024-09-11T05:51:24.654982+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:51:24.654982+0000","last_undegraded":"2024-09-11T05:51:24.654982+0000","last_fullsized":"2024-09-11T05:51:24.654982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":205,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683577+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:51:24.683577+0000","last_peered":"2024-09-11T05:51:24.683577+0000","last_clean":"2024-09-11T05:51:24.683577+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:51:24.683577+0000","last_undegraded":"2024-09-11T05:51:24.683577+0000","last_fullsized":"2024-09-11T05:51:24.683577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"38'42","reported_seq":399,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.718280+0000","last_change":"2024-09-11T05:51:07.507117+0000","last_active":"2024-09-11T05:51:23.718280+0000","last_peered":"2024-09-11T05:51:23.718280+0000","last_clean":"2024-09-11T05:51:23.718280+0000","last_became_active":"2024-09-11T05:51:05.508345+0000","last_became_peered":"2024-09-11T05:51:05.508345+0000","last_unstale":"2024-09-11T05:51:23.718280+0000","last_undegraded":"2024-09-11T05:51:23.718280+0000","last_fullsized":"2024-09-11T05:51:23.718280+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:10:16.718724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":5442445,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":324,"num_read_kb":1939,"num_write":51,"num_write_kb":1395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"40'47","reported_seq":521,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.631163+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:51:24.631163+0000","last_peered":"2024-09-11T05:51:24.631163+0000","last_clean":"2024-09-11T05:51:24.631163+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:51:24.631163+0000","last_undegraded":"2024-09-11T05:51:24.631163+0000","last_fullsized":"2024-09-11T05:51:24.631163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:59.972089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":1899707,"num_objects":10,"num_object_clones":5,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":4,"num_read":448,"num_read_kb":738,"num_write":40,"num_write_kb":242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'111","reported_seq":697,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.803892+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:51:24.803892+0000","last_peered":"2024-09-11T05:51:24.803892+0000","last_clean":"2024-09-11T05:51:24.803892+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:51:24.803892+0000","last_undegraded":"2024-09-11T05:51:24.803892+0000","last_fullsized":"2024-09-11T05:51:24.803892+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:08:04.607370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2387013,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":1387,"num_read_kb":2646,"num_write":119,"num_write_kb":1277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"39'73","reported_seq":440,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-11T05:51:23.932779+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:51:23.932779+0000","last_peered":"2024-09-11T05:51:23.932779+0000","last_clean":"2024-09-11T05:51:23.932779+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:51:23.932779+0000","last_undegraded":"2024-09-11T05:51:23.932779+0000","last_fullsized":"2024-09-11T05:51:23.932779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:21:50.343900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":8810586,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":340,"num_read_kb":4431,"num_write":93,"num_write_kb":3827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"35'12","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.655112+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:51:24.655112+0000","last_peered":"2024-09-11T05:51:24.655112+0000","last_clean":"2024-09-11T05:51:24.655112+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:51:24.655112+0000","last_undegraded":"2024-09-11T05:51:24.655112+0000","last_fullsized":"2024-09-11T05:51:24.655112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:52:18.730528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"38'51","reported_seq":609,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683811+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:51:24.683811+0000","last_peered":"2024-09-11T05:51:24.683811+0000","last_clean":"2024-09-11T05:51:24.683811+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:51:24.683811+0000","last_undegraded":"2024-09-11T05:51:24.683811+0000","last_fullsized":"2024-09-11T05:51:24.683811+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:38:34.205608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":7815960,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":529,"num_read_kb":6088,"num_write":52,"num_write_kb":2131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"40'82","reported_seq":623,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.735520+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:51:24.735520+0000","last_peered":"2024-09-11T05:51:24.735520+0000","last_clean":"2024-09-11T05:51:24.735520+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:51:24.735520+0000","last_undegraded":"2024-09-11T05:51:24.735520+0000","last_fullsized":"2024-09-11T05:51:24.735520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:17:34.042628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":28082230,"num_objects":11,"num_object_clones":5,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":530,"num_read_kb":1265,"num_write":78,"num_write_kb":1194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"40'80","reported_seq":159,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.763157+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:51:24.763157+0000","last_peered":"2024-09-11T05:51:24.763157+0000","last_clean":"2024-09-11T05:51:24.763157+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:51:24.763157+0000","last_undegraded":"2024-09-11T05:51:24.763157+0000","last_fullsized":"2024-09-11T05:51:24.763157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:18.066446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":12539208,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":353,"num_read_kb":7051,"num_write":120,"num_write_kb":3657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"40'51","reported_seq":293,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.638198+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:51:24.638198+0000","last_peered":"2024-09-11T05:51:24.638198+0000","last_clean":"2024-09-11T05:51:24.638198+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:51:24.638198+0000","last_undegraded":"2024-09-11T05:51:24.638198+0000","last_fullsized":"2024-09-11T05:51:24.638198+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:33.440079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9883022,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":220,"num_read_kb":840,"num_write":45,"num_write_kb":380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"38'57","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683535+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:51:24.683535+0000","last_peered":"2024-09-11T05:51:24.683535+0000","last_clean":"2024-09-11T05:51:24.683535+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:51:24.683535+0000","last_undegraded":"2024-09-11T05:51:24.683535+0000","last_fullsized":"2024-09-11T05:51:24.683535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:35.362300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":8696030,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":25,"num_read_kb":179,"num_write":54,"num_write_kb":502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"40'209","reported_seq":510,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.638144+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:51:24.638144+0000","last_peered":"2024-09-11T05:51:24.638144+0000","last_clean":"2024-09-11T05:51:24.638144+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:51:24.638144+0000","last_undegraded":"2024-09-11T05:51:24.638144+0000","last_fullsized":"2024-09-11T05:51:24.638144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:28:59.732626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":3215458,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":1138,"num_read_kb":1824,"num_write":322,"num_write_kb":499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"40'40","reported_seq":92,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-11T05:51:24.683752+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:51:24.683752+0000","last_peered":"2024-09-11T05:51:24.683752+0000","last_clean":"2024-09-11T05:51:24.683752+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:51:24.683752+0000","last_undegraded":"2024-09-11T05:51:24.683752+0000","last_fullsized":"2024-09-11T05:51:24.683752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:08.002000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":1828274,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":3,"num_read_kb":0,"num_write":39,"num_write_kb":496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":210519129,"num_objects":219,"num_object_clones":85,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":51,"num_read":21430,"num_read_kb":78060,"num_write":3469,"num_write_kb":40997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":42029056,"data_stored":71084846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2702,"ondisk_log_size":2702,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40900,"kb_used_data":9712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330940,"statfs":{"total":96636764160,"available":96594882560,"internally_reserved":0,"allocated":9945088,"data_stored":15114952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1572864,"data_stored":2294417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36228,"kb_used_data":3832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335612,"statfs":{"total":96636764160,"available":96599666688,"internally_reserved":0,"allocated":3923968,"data_stored":6461524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44820,"kb_used_data":13636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327020,"statfs":{"total":96636764160,"available":96590868480,"internally_reserved":0,"allocated":13963264,"data_stored":23681207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34196,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337644,"statfs":{"total":96636764160,"available":96601747456,"internally_reserved":0,"allocated":3084288,"data_stored":4641456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29168,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342672,"statfs":{"total":96636764160,"available":96606896128,"internally_reserved":0,"allocated":2187264,"data_stored":2827440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":5197824,"data_stored":8100318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37596,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334244,"statfs":{"total":96636764160,"available":96598265856,"internally_reserved":0,"allocated":6565888,"data_stored":10371223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5545984,"data_stored":9601304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4759552,"data_stored":7920651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1753088,"data_stored":2647800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2650112,"data_stored":4461816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13635584,"data_stored":23588195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3485696,"data_stored":6281865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1273856,"data_stored":2238204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8925184,"data_stored":14345011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:29.505 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T05:51:29.505 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:29.505 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T05:51:29.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T05:51:29.513 INFO:teuthology.orchestra.run.smithi165.stdout:1949 write 0xa1a9788 thru 0xa1b04d0 (0x6d49 bytes) 2024-09-11T05:51:29.522 INFO:teuthology.orchestra.run.smithi165.stdout:1950 write 0x24701fd thru 0x247f95c (0xf760 bytes) 2024-09-11T05:51:29.526 INFO:teuthology.orchestra.run.smithi165.stdout:1951 write 0x7c4588 thru 0x7ca78a (0x6203 bytes) 2024-09-11T05:51:29.526 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:29.526+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:29.530 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.530 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.534 INFO:teuthology.orchestra.run.smithi165.stdout:1952 punch from 0x99a9557 to 0x99b5cb4, (0xc75d bytes) 2024-09-11T05:51:29.534 INFO:teuthology.orchestra.run.smithi165.stdout:1953 punch from 0x604e5a0 to 0x6052225, (0x3c85 bytes) 2024-09-11T05:51:29.534 INFO:teuthology.orchestra.run.smithi165.stdout:1954 read 0x9adae3c thru 0x9ae7468 (0xc62d bytes) 2024-09-11T05:51:29.534 INFO:teuthology.orchestra.run.smithi165.stdout:1955 read 0x62688e3 thru 0x626e637 (0x5d55 bytes) 2024-09-11T05:51:29.534 INFO:teuthology.orchestra.run.smithi165.stdout:1956 write 0x4f5f8be thru 0x4f6df71 (0xe6b4 bytes) 2024-09-11T05:51:29.543 INFO:teuthology.orchestra.run.smithi165.stdout:1957 read 0xa6f6bd thru 0xa7a156 (0xaa9a bytes) 2024-09-11T05:51:29.543 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.543 INFO:teuthology.orchestra.run.smithi165.stdout:1959 read 0x186539b thru 0x1865bb0 (0x816 bytes) 2024-09-11T05:51:29.543 INFO:teuthology.orchestra.run.smithi165.stdout:1960 write 0x5e90d9f thru 0x5e9d833 (0xca95 bytes) 2024-09-11T05:51:29.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:29.557 INFO:teuthology.orchestra.run.smithi165.stdout:1961 trunc from 0xa1b04d1 to 0xe0c072f 2024-09-11T05:51:29.566 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.566 INFO:teuthology.orchestra.run.smithi165.stdout:1963 read 0x8fee6d4 thru 0x8ff14a3 (0x2dd0 bytes) 2024-09-11T05:51:29.566 INFO:teuthology.orchestra.run.smithi165.stdout:1964 write 0x8e58200 thru 0x8e65aa6 (0xd8a7 bytes) 2024-09-11T05:51:29.579 INFO:teuthology.orchestra.run.smithi165.stdout:1965 trunc from 0xe0c072f to 0xa05c340 2024-09-11T05:51:29.601 INFO:teuthology.orchestra.run.smithi165.stdout:1966 write 0xa6b22e4 thru 0xa6c17fe (0xf51b bytes) 2024-09-11T05:51:29.626 INFO:teuthology.orchestra.run.smithi165.stdout:1967 writesame 0xd8e182d thru 0xd8e66d6 data_size 0x1(0x4eaa bytes) 2024-09-11T05:51:29.626 INFO:teuthology.orchestra.run.smithi165.stdout:1968 write 0xa87beda thru 0xa88ba28 (0xfb4f bytes) 2024-09-11T05:51:29.627 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:29.626+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:29.633 INFO:teuthology.orchestra.run.smithi165.stdout:1969 read 0x3d38870 thru 0x3d3eaf7 (0x6288 bytes) 2024-09-11T05:51:29.633 INFO:teuthology.orchestra.run.smithi165.stdout:1970 write 0xd6aa79e thru 0xd6ac0c9 (0x192c bytes) 2024-09-11T05:51:29.649 INFO:teuthology.orchestra.run.smithi165.stdout:1971 trunc from 0xd8e66d7 to 0x43d51e4 2024-09-11T05:51:29.683 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.683 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.683 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:51:29.686 INFO:teuthology.orchestra.run.smithi165.stdout:1972 write 0x9aa6bb6 thru 0x9ab3cbb (0xd106 bytes) 2024-09-11T05:51:29.704 INFO:teuthology.orchestra.run.smithi165.stdout:1973 write 0xa82a372 thru 0xa836f13 (0xcba2 bytes) 2024-09-11T05:51:29.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:29.718 INFO:teuthology.orchestra.run.smithi165.stdout:1974 trunc from 0xa836f14 to 0xe1f275b 2024-09-11T05:51:29.724 INFO:teuthology.orchestra.run.smithi165.stdout:1975 write 0x66a0c70 thru 0x66a56f6 (0x4a87 bytes) 2024-09-11T05:51:29.727 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:29.726+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:29.730 INFO:teuthology.orchestra.run.smithi165.stdout:1977 writesame 0xa5d63a1 thru 0xa5d6f48 data_size 0x4(0xba8 bytes) 2024-09-11T05:51:29.736 INFO:teuthology.orchestra.run.smithi165.stdout:1979 read 0xbc8f35a thru 0xbc9494e (0x55f5 bytes) 2024-09-11T05:51:29.736 INFO:teuthology.orchestra.run.smithi165.stdout:1981 write 0x51cf802 thru 0x51d5102 (0x5901 bytes) 2024-09-11T05:51:29.740 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.740 INFO:teuthology.orchestra.run.smithi165.stdout:1983 write 0x1ee8d7d thru 0x1eec86f (0x3af3 bytes) 2024-09-11T05:51:29.744 INFO:teuthology.orchestra.run.smithi165.stdout:1984 read 0xaeb1936 thru 0xaeb3345 (0x1a10 bytes) 2024-09-11T05:51:29.744 INFO:teuthology.orchestra.run.smithi165.stdout:1985 punch from 0x5ecac4b to 0x5ecb7bb, (0xb70 bytes) 2024-09-11T05:51:29.744 INFO:teuthology.orchestra.run.smithi165.stdout:1987 write 0x7f9cf3a thru 0x7fa9463 (0xc52a bytes) 2024-09-11T05:51:29.750 INFO:teuthology.orchestra.run.smithi165.stdout:1989 read 0xa63b3c2 thru 0xa649358 (0xdf97 bytes) 2024-09-11T05:51:29.751 INFO:teuthology.orchestra.run.smithi165.stdout:1991 writesame 0x73cf39f thru 0x73d5420 data_size 0xb(0x6082 bytes) 2024-09-11T05:51:29.756 INFO:teuthology.orchestra.run.smithi165.stdout:1992 write 0x6caae03 thru 0x6cac3d4 (0x15d2 bytes) 2024-09-11T05:51:29.761 INFO:teuthology.orchestra.run.smithi165.stdout:1993 write 0x48360c4 thru 0x4845299 (0xf1d6 bytes) 2024-09-11T05:51:29.767 INFO:teuthology.orchestra.run.smithi165.stdout:1994 punch from 0x5d0b1e7 to 0x5d17e37, (0xcc50 bytes) 2024-09-11T05:51:29.767 INFO:teuthology.orchestra.run.smithi165.stdout:1995 punch from 0xb5b993 to 0xb6b091, (0xf6fe bytes) 2024-09-11T05:51:29.767 INFO:teuthology.orchestra.run.smithi165.stdout:1996 read 0x52a9241 thru 0x52ae3e5 (0x51a5 bytes) 2024-09-11T05:51:29.784 INFO:teuthology.orchestra.run.smithi165.stdout:1997 trunc from 0xe1f275b to 0x91ee86 2024-09-11T05:51:29.807 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.807 INFO:teuthology.orchestra.run.smithi165.stdout:2000 read 0x5d8ec thru 0x60663 (0x2d78 bytes) 2024-09-11T05:51:29.808 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.819 INFO:teuthology.orchestra.run.smithi165.stdout:2002 write 0x46054c1 thru 0x460cfca (0x7b0a bytes) 2024-09-11T05:51:29.823 INFO:teuthology.orchestra.run.smithi165.stdout:2003 trunc from 0x460cfcb to 0x3085b17 2024-09-11T05:51:29.828 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:29.830+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:29.836 INFO:teuthology.orchestra.run.smithi165.stdout:2004 read 0x44e005 thru 0x450415 (0x2411 bytes) 2024-09-11T05:51:29.836 INFO:teuthology.orchestra.run.smithi165.stdout:2006 punch from 0x9e6b69 to 0x9ea59a, (0x3a31 bytes) 2024-09-11T05:51:29.849 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:29.849 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:29.852 INFO:teuthology.orchestra.run.smithi165.stdout:2007 write 0xb23dc95 thru 0xb247f95 (0xa301 bytes) 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 45, 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "created": "2024-09-11T05:50:35.802404+0000", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "modified": "2024-09-11T05:51:29.684016+0000", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "last_up_change": "2024-09-11T05:50:49.084592+0000", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "last_in_change": "2024-09-11T05:50:38.206686+0000", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_num": 5801984, 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_set": [ 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "noscrub", 2024-09-11T05:51:29.859 INFO:teuthology.orchestra.run.smithi165.stdout: "pglog_hardlimit", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "purged_snapdirs", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_deletes", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "sortbitwise" 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_version": 5, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_snapshot": "", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_max": 3, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "max_osd": 8, 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "require_min_compat_client": "luminous", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "min_compat_client": "jewel", 2024-09-11T05:51:29.860 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "squid", 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "allow_crimson": false, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "pools": [ 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 1, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": ".mgr", 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:50.566112+0000", 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 1, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool", 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:29.861 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 1, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 1, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 1, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 1, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 1, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:29.862 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "18", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 0, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 0, 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:29.863 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:29.864 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:29.865 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "options": { 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_max": 32, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_min": 1 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": {} 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 6.9000000953674316, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 6.9000000953674316, 2024-09-11T05:51:29.866 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.28999999165534973, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 2, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "rbd", 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:54.170388+0000", 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:29.867 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 8, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 8, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 8, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 8, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 8, 2024-09-11T05:51:29.868 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "22", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:29.869 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 2, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 22, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:29.870 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:29.871 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.630000114440918, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.630000114440918, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1, 2024-09-11T05:51:29.872 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2.630000114440918, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2.630000114440918, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "pool_client.0", 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:51:04.331649+0000", 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:29.873 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 32, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 32, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 32, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 32, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 32, 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:29.874 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "44", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:29.875 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 13, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 44, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:29.876 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:29.877 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.878 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 1.5399999618530273, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 1.5399999618530273, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1.1399999856948853, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 1.75, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 1.75, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.879 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "osds": [ 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fdc5357-7501-4582-b301-f5eede8a7bc0", 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 28, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.880 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6815", 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6817", 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.881 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6818", 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6819", 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6822", 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6823", 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6820", 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6821", 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.884 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6817/4049823921", 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6819/4049823921", 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6823/4049823921", 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6821/4049823921", 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:51:29.885 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "78f30980-40a1-46fd-bc1a-a656b32210a9", 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.886 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6800", 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6801", 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6802", 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6803", 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6806", 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6807", 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6804", 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6805", 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:29.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6801/2461301418", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6803/2461301418", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6807/2461301418", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6805/2461301418", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fab9ebc-408e-4bc1-b3ad-385d0e396f6b", 2024-09-11T05:51:29.891 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 19, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.892 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6808", 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6809", 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6810", 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6811", 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6814", 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6816", 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.895 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6812", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6813", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6809/3166670912", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6811/3166670912", 2024-09-11T05:51:29.896 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6816/3166670912", 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6813/3166670912", 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "d3115ede-8d17-4a27-8629-40b19e3c7ee6", 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.897 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6824", 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.898 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6825", 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6826", 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.899 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6827", 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6830", 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.900 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6831", 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6828", 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.901 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6829", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6825/3322140846", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6827/3322140846", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6831/3322140846", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6829/3322140846", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "7fa88502-3ffd-4aac-924b-0d6c65737ac6", 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 28, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.903 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6800", 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6801", 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.904 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6802", 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6803", 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.905 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6806", 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6807", 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.906 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6804", 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6805", 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.907 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6801/347599006", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6803/347599006", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6807/347599006", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6805/347599006", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "03238cca-4d2b-412d-9fcd-f489cb4a8455", 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.908 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6816", 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6817", 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6818", 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6819", 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6822", 2024-09-11T05:51:29.911 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6823", 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.912 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6820", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6821", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6817/4259095655", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6819/4259095655", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6823/4259095655", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6821/4259095655", 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "6c46c2d3-7dd1-4b3a-ad76-391c6e624b40", 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:29.914 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6824", 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6825", 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6826", 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6827", 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.916 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6830", 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6831", 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.917 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6828", 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6829", 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6825/2366978310", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6827/2366978310", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6831/2366978310", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6829/2366978310", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "f849014a-7b82-4361-8a23-3b7a0be5f5c0", 2024-09-11T05:51:29.919 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 28, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.920 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6808", 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6809", 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:29.921 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6810", 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6811", 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6814", 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6815", 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6812", 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6813", 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6809/2576837456", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6811/2576837456", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6815/2576837456", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6813/2576837456", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_xinfo": [ 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.925 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.605278+0000", 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.681171+0000", 2024-09-11T05:51:29.926 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.531762+0000", 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:51:29.927 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:46.158901+0000", 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.928 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.412535+0000", 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.537269+0000", 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.929 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.433446+0000", 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:29.930 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.612887+0000", 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap": [], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_items": [], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_primaries": [], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_temp": [], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_temp": [], 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklist": {}, 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "range_blocklist": {}, 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profiles": { 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "default": { 2024-09-11T05:51:29.931 INFO:teuthology.orchestra.run.smithi165.stdout: "crush-failure-domain": "osd", 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "k": "2", 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "m": "1", 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin": "jerasure", 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "technique": "reed_sol_van" 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps_queue": [], 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "new_removed_snaps": [], 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "new_purged_snaps": [], 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_node_flags": {}, 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "device_class_flags": {}, 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode": { 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_enabled": false, 2024-09-11T05:51:29.932 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_bucket_count": 0, 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout: "degraded_stretch_mode": 0, 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout: "recovering_stretch_mode": 0, 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_bucket": 0 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.933 INFO:teuthology.orchestra.run.smithi165.stdout:2010 write 0x41faadd thru 0x4203103 (0x8627 bytes) 2024-09-11T05:51:29.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:2012 punch from 0x4302ffd to 0x43086be, (0x56c1 bytes) 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:2013 write 0x5f8740d thru 0x5f8becb (0x4abf bytes) 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:2014 write 0xa0761fd thru 0xa07a8a5 (0x46a9 bytes) 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:2016 read 0x45e7ddf thru 0x45ece3e (0x5060 bytes) 2024-09-11T05:51:29.935 INFO:teuthology.orchestra.run.smithi165.stdout:2018 read 0x73df13e thru 0x73e83b2 (0x9275 bytes) 2024-09-11T05:51:29.936 INFO:teuthology.orchestra.run.smithi165.stdout:2019 writesame 0x34994d9 thru 0x349b8bb data_size 0x1(0x23e3 bytes) 2024-09-11T05:51:29.937 INFO:tasks.thrashosds.thrasher:j is {'epoch': 45, 'fsid': 'd5628057-c743-43f5-a51d-1915a16ccfdd', 'created': '2024-09-11T05:50:35.802404+0000', 'modified': '2024-09-11T05:51:29.684016+0000', 'last_up_change': '2024-09-11T05:50:49.084592+0000', 'last_in_change': '2024-09-11T05:50:38.206686+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:50:50.566112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:50:54.170388+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 22, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-11T05:51:04.331649+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 44, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2fdc5357-7501-4582-b301-f5eede8a7bc0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6815', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 4049823921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 4049823921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 4049823921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 4049823921}]}, 'public_addr': '172.21.15.165:6817/4049823921', 'cluster_addr': '172.21.15.165:6819/4049823921', 'heartbeat_back_addr': '172.21.15.165:6823/4049823921', 'heartbeat_front_addr': '172.21.15.165:6821/4049823921', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '78f30980-40a1-46fd-bc1a-a656b32210a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6800', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2461301418}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2461301418}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 2461301418}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 2461301418}]}, 'public_addr': '172.21.15.165:6801/2461301418', 'cluster_addr': '172.21.15.165:6803/2461301418', 'heartbeat_back_addr': '172.21.15.165:6807/2461301418', 'heartbeat_front_addr': '172.21.15.165:6805/2461301418', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3166670912}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3166670912}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6816', 'nonce': 3166670912}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 3166670912}]}, 'public_addr': '172.21.15.165:6809/3166670912', 'cluster_addr': '172.21.15.165:6811/3166670912', 'heartbeat_back_addr': '172.21.15.165:6816/3166670912', 'heartbeat_front_addr': '172.21.15.165:6813/3166670912', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd3115ede-8d17-4a27-8629-40b19e3c7ee6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 3322140846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 3322140846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 3322140846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 3322140846}]}, 'public_addr': '172.21.15.165:6825/3322140846', 'cluster_addr': '172.21.15.165:6827/3322140846', 'heartbeat_back_addr': '172.21.15.165:6831/3322140846', 'heartbeat_front_addr': '172.21.15.165:6829/3322140846', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 347599006}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 347599006}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 347599006}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 347599006}]}, 'public_addr': '172.21.15.172:6801/347599006', 'cluster_addr': '172.21.15.172:6803/347599006', 'heartbeat_back_addr': '172.21.15.172:6807/347599006', 'heartbeat_front_addr': '172.21.15.172:6805/347599006', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03238cca-4d2b-412d-9fcd-f489cb4a8455', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 4259095655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 4259095655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 4259095655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 4259095655}]}, 'public_addr': '172.21.15.172:6817/4259095655', 'cluster_addr': '172.21.15.172:6819/4259095655', 'heartbeat_back_addr': '172.21.15.172:6823/4259095655', 'heartbeat_front_addr': '172.21.15.172:6821/4259095655', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 2366978310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 2366978310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 2366978310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 2366978310}]}, 'public_addr': '172.21.15.172:6825/2366978310', 'cluster_addr': '172.21.15.172:6827/2366978310', 'heartbeat_back_addr': '172.21.15.172:6831/2366978310', 'heartbeat_front_addr': '172.21.15.172:6829/2366978310', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f849014a-7b82-4361-8a23-3b7a0be5f5c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 2576837456}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2576837456}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2576837456}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 2576837456}]}, 'public_addr': '172.21.15.172:6809/2576837456', 'cluster_addr': '172.21.15.172:6811/2576837456', 'heartbeat_back_addr': '172.21.15.172:6815/2576837456', 'heartbeat_front_addr': '172.21.15.172:6813/2576837456', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.605278+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.681171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.531762+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:46.158901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.412535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.537269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.433446+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.612887+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T05:51:29.937 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-11T05:51:29.938 INFO:teuthology.orchestra.run.smithi165.stdout:2021 write 0xda8f874 thru 0xda95fcb (0x6758 bytes) 2024-09-11T05:51:29.938 INFO:teuthology.orchestra.run.smithi165.stdout:2022 read 0xd878c35 thru 0xd87aefe (0x22ca bytes) 2024-09-11T05:51:29.938 INFO:teuthology.orchestra.run.smithi165.stdout:2023 read 0x7162ef1 thru 0x716b314 (0x8424 bytes) 2024-09-11T05:51:29.938 INFO:teuthology.orchestra.run.smithi165.stdout:2024 punch from 0x5313c90 to 0x531b1a2, (0x7512 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2025 write 0x9869709 thru 0x9871f70 (0x8868 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2026 punch from 0xefbd9a to 0xf0844b, (0xc6b1 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2027 punch from 0xa9924a0 to 0xa99aaa8, (0x8608 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2030 trunc from 0xda95fcc to 0xed076da 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2031 write 0x542ec68 thru 0x54333db (0x4774 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2032 read 0x857020c thru 0x857385c (0x3651 bytes) 2024-09-11T05:51:29.939 INFO:teuthology.orchestra.run.smithi165.stdout:2033 writesame 0x74654b9 thru 0x747393d data_size 0x1(0xe485 bytes) 2024-09-11T05:51:29.960 INFO:teuthology.orchestra.run.smithi165.stdout:2034 trunc from 0xed076da to 0x11e1606 2024-09-11T05:51:29.961 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:29.958+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:29.979 INFO:teuthology.orchestra.run.smithi165.stdout:2035 read 0xb7a08e thru 0xb8582e (0xb7a1 bytes) 2024-09-11T05:51:29.993 INFO:teuthology.orchestra.run.smithi165.stdout:2036 trunc from 0x11e1606 to 0xd1216ca 2024-09-11T05:51:30.002 INFO:teuthology.orchestra.run.smithi165.stdout:2037 write 0x9a26a61 thru 0x9a30351 (0x98f1 bytes) 2024-09-11T05:51:30.008 INFO:teuthology.orchestra.run.smithi165.stdout:2038 writesame 0x6c92c4a thru 0x6ca00b9 data_size 0xc(0xd470 bytes) 2024-09-11T05:51:30.016 INFO:teuthology.orchestra.run.smithi165.stdout:2040 write 0xb70cdac thru 0xb7115e7 (0x483c bytes) 2024-09-11T05:51:30.020 INFO:teuthology.orchestra.run.smithi165.stdout:2041 read 0x2024e87 thru 0x2028379 (0x34f3 bytes) 2024-09-11T05:51:30.020 INFO:teuthology.orchestra.run.smithi165.stdout:2043 read 0x4348d74 thru 0x4350d26 (0x7fb3 bytes) 2024-09-11T05:51:30.033 INFO:teuthology.orchestra.run.smithi165.stdout:2047 trunc from 0xd1216ca to 0x21efe17 2024-09-11T05:51:30.039 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:30.038+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:30.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:30.063 INFO:teuthology.orchestra.run.smithi165.stdout:2048 writesame 0x3b07a8e thru 0x3b07ce7 data_size 0x7(0x25a bytes) 2024-09-11T05:51:30.074 INFO:teuthology.orchestra.run.smithi165.stdout:2050 writesame 0x42136ab thru 0x4222d52 data_size 0x24(0xf6a8 bytes) 2024-09-11T05:51:30.074 INFO:teuthology.orchestra.run.smithi165.stdout:2051 read 0x66af43 thru 0x67251b (0x75d9 bytes) 2024-09-11T05:51:30.074 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.077 INFO:teuthology.orchestra.run.smithi165.stdout:2053 trunc from 0x4222d53 to 0x6bb95c8 2024-09-11T05:51:30.090 INFO:teuthology.orchestra.run.smithi165.stdout:2054 read 0x43dc759 thru 0x43df93a (0x31e2 bytes) 2024-09-11T05:51:30.090 INFO:teuthology.orchestra.run.smithi165.stdout:2056 read 0x5b09548 thru 0x5b144d2 (0xaf8b bytes) 2024-09-11T05:51:30.090 INFO:teuthology.orchestra.run.smithi165.stdout:2057 punch from 0x130098b to 0x13069bf, (0x6034 bytes) 2024-09-11T05:51:30.091 INFO:teuthology.orchestra.run.smithi165.stdout:2059 trunc from 0x6bb95c8 to 0x7e549c3 2024-09-11T05:51:30.092 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:30.092 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:30.101 INFO:teuthology.orchestra.run.smithi165.stdout:2060 writesame 0x4b931d5 thru 0x4b99c86 data_size 0x7(0x6ab2 bytes) 2024-09-11T05:51:30.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:30.110 INFO:teuthology.orchestra.run.smithi165.stdout:2061 punch from 0x3572687 to 0x357ae9e, (0x8817 bytes) 2024-09-11T05:51:30.110 INFO:teuthology.orchestra.run.smithi165.stdout:2062 read 0x6fd82de thru 0x6fe1b32 (0x9855 bytes) 2024-09-11T05:51:30.112 INFO:teuthology.orchestra.run.smithi165.stdout:2063 trunc from 0x7e549c3 to 0x99e9a85 2024-09-11T05:51:30.131 INFO:teuthology.orchestra.run.smithi165.stdout:2064 write 0xe37cd62 thru 0xe38bdaa (0xf049 bytes) 2024-09-11T05:51:30.138 INFO:teuthology.orchestra.run.smithi165.stdout:2065 read 0x267b970 thru 0x268b5b1 (0xfc42 bytes) 2024-09-11T05:51:30.139 INFO:teuthology.orchestra.run.smithi165.stdout:2066 punch from 0xb5af40 to 0xb65d46, (0xae06 bytes) 2024-09-11T05:51:30.139 INFO:teuthology.orchestra.run.smithi165.stdout:2067 read 0x13eb7a7 thru 0x13f1595 (0x5def bytes) 2024-09-11T05:51:30.139 INFO:teuthology.orchestra.run.smithi165.stdout:2068 write 0x1555cd9 thru 0x1561458 (0xb780 bytes) 2024-09-11T05:51:30.139 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:30.138+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:30.144 INFO:teuthology.orchestra.run.smithi165.stdout:2069 punch from 0x3edaccd to 0x3edb5ef, (0x922 bytes) 2024-09-11T05:51:30.145 INFO:teuthology.orchestra.run.smithi165.stdout:2072 write 0x81f0dd8 thru 0x81fa44f (0x9678 bytes) 2024-09-11T05:51:30.151 INFO:teuthology.orchestra.run.smithi165.stdout:2073 write 0xa88d174 thru 0xa89aa25 (0xd8b2 bytes) 2024-09-11T05:51:30.156 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.156 INFO:teuthology.orchestra.run.smithi165.stdout:2077 writesame 0xdc08f17 thru 0xdc146ca data_size 0x6(0xb7b4 bytes) 2024-09-11T05:51:30.173 INFO:teuthology.orchestra.run.smithi165.stdout:2078 trunc from 0xe38bdab to 0xc8de56b 2024-09-11T05:51:30.189 INFO:teuthology.orchestra.run.smithi165.stdout:2082 read 0xb01c01e thru 0xb021a51 (0x5a34 bytes) 2024-09-11T05:51:30.190 INFO:teuthology.orchestra.run.smithi165.stdout:2083 write 0x31d0c80 thru 0x31e053b (0xf8bc bytes) 2024-09-11T05:51:30.195 INFO:teuthology.orchestra.run.smithi165.stdout:2084 write 0x54e561a thru 0x54f392d (0xe314 bytes) 2024-09-11T05:51:30.199 INFO:teuthology.orchestra.run.smithi165.stdout:2085 write 0xb52c2df thru 0xb5361ff (0x9f21 bytes) 2024-09-11T05:51:30.205 INFO:teuthology.orchestra.run.smithi165.stdout:2086 read 0x22f30a thru 0x23da5f (0xe756 bytes) 2024-09-11T05:51:30.206 INFO:teuthology.orchestra.run.smithi165.stdout:2087 punch from 0x7576c7e to 0x75834e1, (0xc863 bytes) 2024-09-11T05:51:30.206 INFO:teuthology.orchestra.run.smithi165.stdout:2088 read 0x19fbb97 thru 0x19fc488 (0x8f2 bytes) 2024-09-11T05:51:30.206 INFO:teuthology.orchestra.run.smithi165.stdout:2090 write 0x7f4b3bd thru 0x7f596d4 (0xe318 bytes) 2024-09-11T05:51:30.210 INFO:teuthology.orchestra.run.smithi165.stdout:2091 writesame 0x451107 thru 0x45df45 data_size 0x1(0xce3f bytes) 2024-09-11T05:51:30.223 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:30.224 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:30.240 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:30.242+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:30.246 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:30.246 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:30.253 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:30.253 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:30.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:30.270 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:30.270 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:30.283 INFO:teuthology.orchestra.run.smithi165.stdout:2092 write 0xb0b4048 thru 0xb0b8e2b (0x4de4 bytes) 2024-09-11T05:51:30.287 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:30.287 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:30.288 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.288 INFO:teuthology.orchestra.run.smithi165.stdout:2096 punch from 0xbfa3935 to 0xbfb2044, (0xe70f bytes) 2024-09-11T05:51:30.297 INFO:teuthology.orchestra.run.smithi165.stdout:2098 trunc from 0xc8de56b to 0x672138d 2024-09-11T05:51:30.314 INFO:teuthology.orchestra.run.smithi165.stdout:2099 punch from 0x2c4091d to 0x2c48764, (0x7e47 bytes) 2024-09-11T05:51:30.317 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:30.331 INFO:teuthology.orchestra.run.smithi165.stdout:2100 write 0xe6c5637 thru 0xe6c98ee (0x42b8 bytes) 2024-09-11T05:51:30.341 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:30.342+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:30.342 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:30.342 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:30.343 INFO:teuthology.orchestra.run.smithi165.stdout:2101 trunc from 0xe6c98ef to 0x97a2952 2024-09-11T05:51:30.364 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:30.364 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:30.369 INFO:teuthology.orchestra.run.smithi165.stdout:2102 write 0xd1415ff thru 0xd145056 (0x3a58 bytes) 2024-09-11T05:51:30.384 INFO:teuthology.orchestra.run.smithi165.stdout:2103 trunc from 0xd145057 to 0x6681887 2024-09-11T05:51:30.390 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:30.390 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:30.402 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:30.402 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:30.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6094 3.4 3:25de1119:::rbd_data.11249b359231.0000000000000004:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.040416124, 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011224494999999999, 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6094 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.403 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362468+0000", 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362469+0000", 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-06 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362477+0000", 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362496+0000", 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8598999999999999e-05 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362506+0000", 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0336999999999999e-05 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362576+0000", 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0401000000000006e-05 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362671+0000", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4110999999999994e-05 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364415+0000", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017441049999999999 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373558+0000", 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091431189999999999 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373596+0000", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.803e-05 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373637+0000", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0706e-05 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.373692+0000", 2024-09-11T05:51:30.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5531000000000001e-05 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:11133 3.0 3:061d780f:::rbd_data.1124518b2c12.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3743510000000008, 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03149018, 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 11133 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028533+0000", 2024-09-11T05:51:30.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028534+0000", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7850000000000001e-06 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028542+0000", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8879999999999997e-06 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028559+0000", 2024-09-11T05:51:30.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6603999999999999e-05 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028567+0000", 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9019999999999999e-06 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028635+0000", 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8046999999999994e-05 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.028805+0000", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016979000000000001 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.033986+0000", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051812589999999997 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059864+0000", 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025877539000000001 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059924+0000", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0327e-05 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059971+0000", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7239e-05 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060023+0000", 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1799999999999999e-05 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.414 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:11133 3.0 e36/23 ondisk, result = 0)", 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3543180239999995, 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011468028, 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:30.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048566+0000", 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048567+0000", 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.19e-06 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048573+0000", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9290000000000003e-06 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048591+0000", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8179999999999999e-05 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.048601+0000", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8300000000000008e-06 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059910+0000", 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011309476000000001 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.059919+0000", 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9660000000000002e-06 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:21.060034+0000", 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011445700000000001 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14816 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7709523350000005, 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010070437, 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14816 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631931+0000", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631932+0000", 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4499999999999998e-07 2024-09-11T05:51:30.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631934+0000", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.136e-06 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631940+0000", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5679999999999996e-06 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631946+0000", 2024-09-11T05:51:30.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4349999999999996e-06 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641825+0000", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098788680000000007 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.641897+0000", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2577999999999995e-05 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642002+0000", 2024-09-11T05:51:30.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010450699999999999 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14824 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7709290490000003, 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011351063, 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14824 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:30.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631955+0000", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.58e-07 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631957+0000", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4169999999999999e-06 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631965+0000", 2024-09-11T05:51:30.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6149999999999999e-06 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631971+0000", 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7570000000000001e-06 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642045+0000", 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010074112 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642124+0000", 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9067999999999996e-05 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643306+0000", 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011817360000000001 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14825 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 3670016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.427 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.770905537, 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010650197, 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14825 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631978+0000", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631979+0000", 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2199999999999999e-07 2024-09-11T05:51:30.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631981+0000", 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.176e-06 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631988+0000", 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4490000000000002e-06 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631994+0000", 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.507e-06 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642464+0000", 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010470187000000001 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642540+0000", 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6058999999999998e-05 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642628+0000", 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8397000000000001e-05 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14833 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.770882705, 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010835374, 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.432 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14833 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632001+0000", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0999999999999999e-07 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632003+0000", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0600000000000002e-06 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632011+0000", 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2790000000000001e-06 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632016+0000", 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5450000000000003e-06 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642656+0000", 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010639414999999999 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642728+0000", 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2166e-05 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642836+0000", 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010849900000000001 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14834 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.436 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7708601169999998, 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011001664, 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14834 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632024+0000", 2024-09-11T05:51:30.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632026+0000", 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632035+0000", 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.217e-06 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632041+0000", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5289999999999999e-06 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642862+0000", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010820689 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.642935+0000", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3579999999999997e-05 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643025+0000", 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0070999999999999e-05 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14835 3.0 3:0075ed35:::rbd_data.1124ef904871.0000000000000008:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7708348760000003, 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011160765, 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14835 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632049+0000", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9799999999999999e-07 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632052+0000", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4480000000000001e-06 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632060+0000", 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3259999999999996e-06 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632068+0000", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6830000000000008e-06 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643051+0000", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010983153000000001 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643125+0000", 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3967000000000005e-05 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643210+0000", 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4790000000000006e-05 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14844 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7708079310000002, 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011642978999999999, 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.445 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14844 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632076+0000", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632082+0000", 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632095+0000", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3454e-05 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632102+0000", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1430000000000003e-06 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643234+0000", 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011132918 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643339+0000", 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104734 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643719+0000", 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000379551 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14845 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4456448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.449 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7707707450000001, 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011793567, 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14845 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:30.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632113+0000", 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5299999999999999e-07 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632116+0000", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3039999999999999e-06 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632124+0000", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6640000000000004e-06 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632127+0000", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.948e-06 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643806+0000", 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011678932 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643856+0000", 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0198999999999998e-05 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.643907+0000", 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0067e-05 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14846 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7707526199999997, 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01202668, 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14846 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.454 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632131+0000", 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.23e-07 2024-09-11T05:51:30.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632138+0000", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2559999999999999e-06 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632143+0000", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0370000000000002e-06 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632146+0000", 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5489999999999998e-06 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644063+0000", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011917116 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644109+0000", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5676000000000001e-05 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644158+0000", 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8823000000000003e-05 2024-09-11T05:51:30.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14847 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.770733667, 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012106896000000001, 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14847 2024-09-11T05:51:30.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632150+0000", 2024-09-11T05:51:30.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.91e-07 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632152+0000", 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.767e-06 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632158+0000", 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632162+0000", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644175+0000", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012013835 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644214+0000", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8216999999999997e-05 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644257+0000", 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3257999999999998e-05 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14874 3.0 3:040978a7:::rbd_data.1124ef904871.0000000000000004:a [sparse-read 3407872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7707158310000004, 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012189050999999999, 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.462 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14874 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632168+0000", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5799999999999999e-07 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632170+0000", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0310000000000001e-06 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632176+0000", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7780000000000004e-06 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632181+0000", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3710000000000002e-06 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644274+0000", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012093303999999999 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644315+0000", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0683000000000003e-05 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644357+0000", 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2326000000000001e-05 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14875 3.0 3:040978a7:::rbd_data.1124ef904871.0000000000000004:a [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7706930290000003, 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012260057, 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.466 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14875 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632191+0000", 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8799999999999998e-07 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632193+0000", 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1969999999999999e-06 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632200+0000", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4139999999999999e-06 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632205+0000", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8380000000000001e-06 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644372+0000", 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012167585 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644410+0000", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7462000000000001e-05 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644451+0000", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1173e-05 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14878 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:30.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7706747519999997, 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012364757000000001, 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14878 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632209+0000", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5499999999999999e-07 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632211+0000", 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.328e-06 2024-09-11T05:51:30.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632214+0000", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6169999999999998e-06 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632223+0000", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3960000000000006e-06 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644467+0000", 2024-09-11T05:51:30.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012244147 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644527+0000", 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0186000000000003e-05 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644574+0000", 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6828e-05 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.474 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14879 3.0 3:0661515f:::rbd_data.1124ef904871.000000000000000b:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.770654478, 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012781137, 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14879 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:30.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632229+0000", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632230+0000", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632232+0000", 2024-09-11T05:51:30.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0930000000000001e-06 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632238+0000", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2840000000000003e-06 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.632243+0000", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5809999999999998e-06 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644660+0000", 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012417901 2024-09-11T05:51:30.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644879+0000", 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021857800000000001 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645010+0000", 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013134400000000001 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.478 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:15154 3.19 3:9b307830:::rbd_data.11243bbd48bb.0000000000000039:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.5017485639999997, 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016613869999999999, 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 15154 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901135+0000", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901138+0000", 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4329999999999998e-06 2024-09-11T05:51:30.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901148+0000", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0553e-05 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901168+0000", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0360000000000002e-05 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901177+0000", 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7660000000000004e-06 2024-09-11T05:51:30.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901251+0000", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3504999999999995e-05 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.901344+0000", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3381999999999995e-05 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.904163+0000", 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028184880000000001 2024-09-11T05:51:30.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917590+0000", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013427346999999999 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917644+0000", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3659000000000002e-05 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917700+0000", 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5995000000000002e-05 2024-09-11T05:51:30.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.917749+0000", 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9382000000000003e-05 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:16467 3.0 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:b [read 442368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.792346+0000", 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.6105380899999999, 2024-09-11T05:51:30.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01006604, 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 16467 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792346+0000", 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792346+0000", 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792346+0000", 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4000000000000002e-07 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792348+0000", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.277e-06 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792355+0000", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3289999999999999e-06 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792360+0000", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802283+0000", 2024-09-11T05:51:30.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099230689999999996 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802344+0000", 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1464000000000006e-05 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802412+0000", 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7174000000000004e-05 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:16468 3.0 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:b [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.792367+0000", 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.6105168480000001, 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010370247000000001, 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 16468 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.489 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792367+0000", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792367+0000", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792367+0000", 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7500000000000001e-07 2024-09-11T05:51:30.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792369+0000", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2139999999999999e-06 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792375+0000", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9880000000000001e-06 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.792380+0000", 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.442e-06 2024-09-11T05:51:30.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802495+0000", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010114654000000001 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802629+0000", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013427299999999999 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.802737+0000", 2024-09-11T05:51:30.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010830100000000001 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout:2104 read 0x10251d4 thru 0x10316a9 (0xc4d6 bytes) 2024-09-11T05:51:30.493 INFO:teuthology.orchestra.run.smithi165.stdout:2105 trunc from 0x6681887 to 0x9fd3a67 2024-09-11T05:51:30.494 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.494 INFO:teuthology.orchestra.run.smithi165.stdout:2108 write 0x46d2145 thru 0x46d48b1 (0x276d bytes) 2024-09-11T05:51:30.495 INFO:teuthology.orchestra.run.smithi165.stdout:2109 punch from 0x60754eb to 0x6077c2f, (0x2744 bytes) 2024-09-11T05:51:30.495 INFO:teuthology.orchestra.run.smithi165.stdout:2110 writesame 0x8664720 thru 0x8668de4 data_size 0x1b(0x46c5 bytes) 2024-09-11T05:51:30.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2112 trunc from 0x9fd3a67 to 0x3eb3d7a 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2113 write 0x54091e1 thru 0x5418632 (0xf452 bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2114 punch from 0x24abd03 to 0x24b1520, (0x581d bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2115 writesame 0x8e44107 thru 0x8e50fcd data_size 0x5(0xcec7 bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2116 read 0x6de91b6 thru 0x6de9732 (0x57d bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2118 writesame 0x7a5a03e thru 0x7a5e3b5 data_size 0x22(0x4378 bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2119 read 0x2de9740 thru 0x2dedd9d (0x465e bytes) 2024-09-11T05:51:30.497 INFO:teuthology.orchestra.run.smithi165.stdout:2120 write 0x7188329 thru 0x718f198 (0x6e70 bytes) 2024-09-11T05:51:30.498 INFO:teuthology.orchestra.run.smithi165.stdout:2122 write 0xbbdc98d thru 0xbbea346 (0xd9ba bytes) 2024-09-11T05:51:30.498 INFO:teuthology.orchestra.run.smithi165.stdout:2123 write 0xa1f02cc thru 0xa1f7198 (0x6ecd bytes) 2024-09-11T05:51:30.498 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:30.494+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:30.509 INFO:teuthology.orchestra.run.smithi165.stdout:2124 writesame 0xca70b93 thru 0xca7e75c data_size 0x7(0xdbca bytes) 2024-09-11T05:51:30.509 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.509 INFO:teuthology.orchestra.run.smithi165.stdout:2129 read 0x4bb9120 thru 0x4bc50c5 (0xbfa6 bytes) 2024-09-11T05:51:30.509 INFO:teuthology.orchestra.run.smithi165.stdout:2131 write 0x7b318d3 thru 0x7b3bf1b (0xa649 bytes) 2024-09-11T05:51:30.513 INFO:teuthology.orchestra.run.smithi165.stdout:2132 punch from 0x6a886b0 to 0x6a94512, (0xbe62 bytes) 2024-09-11T05:51:30.513 INFO:teuthology.orchestra.run.smithi165.stdout:2133 write 0x8f873eb thru 0x8f8c06e (0x4c84 bytes) 2024-09-11T05:51:30.517 INFO:teuthology.orchestra.run.smithi165.stdout:2134 read 0x3cb78c7 thru 0x3cba5d6 (0x2d10 bytes) 2024-09-11T05:51:30.517 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.517 INFO:teuthology.orchestra.run.smithi165.stdout:2139 writesame 0x25fe3a9 thru 0x25fec72 data_size 0x1(0x8ca bytes) 2024-09-11T05:51:30.522 INFO:teuthology.orchestra.run.smithi165.stdout:2140 read 0xb046b9d thru 0xb04f2b7 (0x871b bytes) 2024-09-11T05:51:30.522 INFO:teuthology.orchestra.run.smithi165.stdout:2142 read 0x89ccf2 thru 0x89f3db (0x26ea bytes) 2024-09-11T05:51:30.522 INFO:teuthology.orchestra.run.smithi165.stdout:2143 write 0x2cafb39 thru 0x2cb08f1 (0xdb9 bytes) 2024-09-11T05:51:30.527 INFO:teuthology.orchestra.run.smithi165.stdout:2144 punch from 0x5da2d17 to 0x5db23bf, (0xf6a8 bytes) 2024-09-11T05:51:30.541 INFO:teuthology.orchestra.run.smithi165.stdout:2145 trunc from 0xca7e75d to 0x1a5523d 2024-09-11T05:51:30.557 INFO:teuthology.orchestra.run.smithi165.stdout:2146 write 0x1300af5 thru 0x130e162 (0xd66e bytes) 2024-09-11T05:51:30.562 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.570 INFO:teuthology.orchestra.run.smithi165.stdout:2148 write 0x3053148 thru 0x30570fa (0x3fb3 bytes) 2024-09-11T05:51:30.575 INFO:teuthology.orchestra.run.smithi165.stdout:2149 punch from 0xc9323a to 0xc9c71c, (0x94e2 bytes) 2024-09-11T05:51:30.575 INFO:teuthology.orchestra.run.smithi165.stdout:2150 punch from 0x2ab5694 to 0x2ac4e1a, (0xf786 bytes) 2024-09-11T05:51:30.575 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:30.575 INFO:teuthology.orchestra.run.smithi165.stdout:2152 read 0x2fc3ef thru 0x3068c0 (0xa4d2 bytes) 2024-09-11T05:51:30.576 INFO:teuthology.orchestra.run.smithi165.stdout:2155 read 0x3ce7f9 thru 0x3d0ffa (0x2802 bytes) 2024-09-11T05:51:30.576 INFO:teuthology.orchestra.run.smithi165.stdout:2158 clone 12 order 20 su 8192 sc 11 2024-09-11T05:51:30.598 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:30.597+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:30.618 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:30.618 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:30.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:30.719 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:30.718+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:30.800 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:30.798+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:30.804 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:30.804 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:30.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:30.857 INFO:teuthology.orchestra.run.smithi165.stdout:leaving image image_client.0-clone11 intact 2024-09-11T05:51:30.901 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:30.901+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.773581630000001, 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:30.987 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:30.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6060 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'29, mlcod=33'27)", 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.671270995, 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024674810000000001, 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314248+0000", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4099999999999997e-07 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314255+0000", 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314269+0000", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4177e-05 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314277+0000", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6279999999999996e-06 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314341+0000", 2024-09-11T05:51:30.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3539000000000003e-05 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314363+0000", 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2272e-05 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316606+0000", 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002243629 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:30.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.316715+0000", 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108576 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.670591899, 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:30.996 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:30.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:30.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:31.000 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.576049863, 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:31.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:31.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.304302579, 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:31.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.009 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.180283605, 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6354 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'25, mlcod=33'23)", 2024-09-11T05:51:31.014 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.168910433000001, 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026120029999999999, 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816609+0000", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816617+0000", 2024-09-11T05:51:31.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7680000000000008e-06 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816630+0000", 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816637+0000", 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:31.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816690+0000", 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2533e-05 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816706+0000", 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6407000000000001e-05 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819170+0000", 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024645309999999998 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819220+0000", 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9595999999999997e-05 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.019 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.105291428999999, 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:31.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6672 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'39, mlcod=33'37)", 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.686737831, 2024-09-11T05:51:31.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022554760000000002, 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298781+0000", 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298783+0000", 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.362e-06 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298845+0000", 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1544999999999994e-05 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298878+0000", 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3642000000000001e-05 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.298895+0000", 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6750000000000001e-05 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299066+0000", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017087600000000001 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.299124+0000", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8213e-05 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.300980+0000", 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00185586 2024-09-11T05:51:31.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.301036+0000", 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6227999999999998e-05 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.631316309000001, 2024-09-11T05:51:31.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:31.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.138687044999999, 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:31.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7116 3.9 e34/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 34'35, mlcod=34'33)", 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.978459067999999, 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00371405, 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007061+0000", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5090000000000001e-06 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007072+0000", 2024-09-11T05:51:31.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1399e-05 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007100+0000", 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7531000000000001e-05 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007114+0000", 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4378e-05 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007340+0000", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022568499999999999 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007387+0000", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7194000000000002e-05 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010717+0000", 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033296369999999999 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010773+0000", 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6716999999999999e-05 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7119 3.4 e34/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 34'49, mlcod=34'47)", 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.97725329, 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034162110000000002, 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.043 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008267+0000", 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.294e-06 2024-09-11T05:51:31.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008276+0000", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5820000000000008e-06 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008290+0000", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4005000000000001e-05 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008298+0000", 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3189999999999995e-06 2024-09-11T05:51:31.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008368+0000", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0055999999999995e-05 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008396+0000", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7563999999999999e-05 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011628+0000", 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032314969999999998 2024-09-11T05:51:31.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011681+0000", 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3894000000000003e-05 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12734 3.0 e38/23 3:06e59eda:::rbd_data.1124518b2c12.000000000000002b:head v 38'41, mlcod=37'39)", 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.1752432870000007, 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033376479999999999, 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.047 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810277+0000", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810314+0000", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7370999999999998e-05 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810336+0000", 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2129000000000001e-05 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810347+0000", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0465999999999999e-05 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810418+0000", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099000000000005e-05 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810443+0000", 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4825999999999999e-05 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813553+0000", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031099790000000001 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813613+0000", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0307999999999999e-05 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15206 3.0 e41/23 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:head v 41'65, mlcod=41'63)", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:31.051 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.9815430559999996, 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034631129999999999, 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003977+0000", 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003986+0000", 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004008+0000", 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1679e-05 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004018+0000", 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5210000000000008e-06 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004105+0000", 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7371000000000001e-05 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004128+0000", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2682999999999999e-05 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007395+0000", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032676129999999999 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007439+0000", 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3306999999999997e-05 2024-09-11T05:51:31.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15537 3.4 e42/23 3:242e3831:::rbd_data.11243bbd48bb.000000000000001e:head v 42'64, mlcod=41'61)", 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2224237069999999, 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031185620000000001, 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763096+0000", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763105+0000", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2119999999999992e-06 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763125+0000", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9536999999999999e-05 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763135+0000", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0022e-05 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763216+0000", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1606000000000004e-05 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763236+0000", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9585e-05 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766184+0000", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002948169 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766213+0000", 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.917e-05 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15715 3.0 e42/23 3:0151ec39:::rbd_data.1124d6faf06d.000000000000006a:head v 42'71, mlcod=42'67)", 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.0657151130000004, 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028945469999999999, 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.060 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919804+0000", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-07 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919810+0000", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8919999999999997e-06 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919823+0000", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2879e-05 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919830+0000", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.635e-06 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919877+0000", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7049999999999998e-05 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919896+0000", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9216999999999999e-05 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922658+0000", 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002762675 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922698+0000", 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9477000000000002e-05 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:16482 3.9 e43/23 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:head v 43'50, mlcod=40'48)", 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1819783749999999, 2024-09-11T05:51:31.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046884810000000004, 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803541+0000", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8899999999999999e-07 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.804985+0000", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014445459999999999 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805024+0000", 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8390999999999999e-05 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805036+0000", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799e-05 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805107+0000", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1530000000000001e-05 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805135+0000", 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8384000000000001e-05 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808057+0000", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029216979999999999 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808229+0000", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017144400000000001 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:16483 3.9 e43/23 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:head v 43'52, mlcod=40'48)", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.806329+0000", 2024-09-11T05:51:31.068 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1791896990000001, 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023325500000000001, 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806329+0000", 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806329+0000", 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806330+0000", 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1820000000000001e-06 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806341+0000", 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1065e-05 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806358+0000", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7031e-05 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806372+0000", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3777e-05 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806632+0000", 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000260043 2024-09-11T05:51:31.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.806663+0000", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1513999999999997e-05 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808625+0000", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019613159999999998 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808661+0000", 2024-09-11T05:51:31.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6622000000000003e-05 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:18436 3.0 e45/23 3:05af81b0:::rbd_data.112453064ca6.0000000000000085:head v 45'158, mlcod=45'156)", 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:30.423077+0000", 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.56244104800000005, 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022613590000000001, 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423077+0000", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423077+0000", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423079+0000", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4920000000000001e-06 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423094+0000", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4715000000000001e-05 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423110+0000", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6215e-05 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423118+0000", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4989999999999994e-06 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423176+0000", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8044999999999997e-05 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.423200+0000", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3906000000000001e-05 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.425309+0000", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002108558 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:30.425339+0000", 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9929000000000001e-05 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.077 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:31.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:31.078 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-09-11T05:51:31.079 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:31.078+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:31.171 INFO:teuthology.orchestra.run.smithi165.stdout:2159 punch from 0x1bb5c69 to 0x1bc1da4, (0xc13b bytes) 2024-09-11T05:51:31.179 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:31.178+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:31.180 INFO:teuthology.orchestra.run.smithi165.stdout:2160 write 0x26ec2e8 thru 0x26f9f31 (0xdc4a bytes) 2024-09-11T05:51:31.206 INFO:teuthology.orchestra.run.smithi165.stdout:2161 read 0x1919100 thru 0x191d842 (0x4743 bytes) 2024-09-11T05:51:31.206 INFO:teuthology.orchestra.run.smithi165.stdout:2162 punch from 0xd3306 to 0xdf4d8, (0xc1d2 bytes) 2024-09-11T05:51:31.206 INFO:teuthology.orchestra.run.smithi165.stdout:2163 read 0x210c36f thru 0x21161fc (0x9e8e bytes) 2024-09-11T05:51:31.207 INFO:teuthology.orchestra.run.smithi165.stdout:2165 write 0x12a4029 thru 0x12aa780 (0x6758 bytes) 2024-09-11T05:51:31.216 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19242 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212478+0000", 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0039849029999999997, 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040030380000000004, 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19242 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212478+0000", 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212478+0000", 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212479+0000", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2899999999999999e-06 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212486+0000", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4570000000000002e-06 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212504+0000", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8159e-05 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212518+0000", 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4491999999999999e-05 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19243 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212567+0000", 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0038961450000000002, 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039589619999999999, 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19243 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212567+0000", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212567+0000", 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212568+0000", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.567e-06 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212575+0000", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8560000000000001e-06 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212596+0000", 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0928999999999999e-05 2024-09-11T05:51:31.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212611+0000", 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19253 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212881+0000", 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.003581716, 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003674386, 2024-09-11T05:51:31.223 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19253 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212881+0000", 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212881+0000", 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212882+0000", 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212888+0000", 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0469999999999999e-06 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212906+0000", 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8138000000000002e-05 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212920+0000", 2024-09-11T05:51:31.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.329e-05 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19254 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212937+0000", 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0035261630000000001, 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036602359999999999, 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19254 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212937+0000", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212937+0000", 2024-09-11T05:51:31.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212938+0000", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1969999999999999e-06 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212943+0000", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2299999999999999e-06 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212961+0000", 2024-09-11T05:51:31.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7659000000000002e-05 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212974+0000", 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3182e-05 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19255 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213010+0000", 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0034525939999999998, 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00364205, 2024-09-11T05:51:31.230 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19255 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213010+0000", 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213010+0000", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213011+0000", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213017+0000", 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6289999999999998e-06 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213036+0000", 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8476000000000001e-05 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213049+0000", 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3129000000000001e-05 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19256 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213066+0000", 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0033972080000000001, 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036466039999999999, 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19256 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213066+0000", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213066+0000", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213066+0000", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0699999999999996e-07 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213070+0000", 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.703e-06 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213081+0000", 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0523000000000001e-05 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213088+0000", 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6320000000000008e-06 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19257 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213098+0000", 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0033650759999999998, 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036687030000000002, 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19257 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213098+0000", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213098+0000", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213098+0000", 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8599999999999997e-07 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213101+0000", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.977e-06 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213111+0000", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4720000000000001e-06 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213120+0000", 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7299999999999994e-06 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.239 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19258 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213132+0000", 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.003330689, 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036677049999999998, 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19258 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213132+0000", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213132+0000", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213133+0000", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3e-07 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213136+0000", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4460000000000001e-06 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213146+0000", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7899999999999994e-06 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213153+0000", 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9029999999999998e-06 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19259 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213162+0000", 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0033006670000000002, 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003671863, 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19259 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213162+0000", 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213162+0000", 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213163+0000", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213170+0000", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0060000000000003e-06 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213179+0000", 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6320000000000003e-06 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213187+0000", 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2629999999999997e-06 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19260 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213203+0000", 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00325939, 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00366826, 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19260 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213203+0000", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213203+0000", 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213204+0000", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213209+0000", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0560000000000004e-06 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213223+0000", 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3623e-05 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213230+0000", 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.306e-06 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19261 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213244+0000", 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0032187100000000001, 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036736830000000001, 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.249 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19261 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213244+0000", 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213244+0000", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213245+0000", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1739999999999999e-06 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213250+0000", 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0730000000000004e-06 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213267+0000", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7102e-05 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213281+0000", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3623e-05 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19262 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213297+0000", 2024-09-11T05:51:31.252 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0031654560000000001, 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036652099999999999, 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19262 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213297+0000", 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213297+0000", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213298+0000", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999991e-07 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213303+0000", 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9089999999999999e-06 2024-09-11T05:51:31.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213320+0000", 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6461999999999999e-05 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213333+0000", 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3203e-05 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19263 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.213349+0000", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0031137769999999999, 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036532190000000001, 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19263 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213349+0000", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213349+0000", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213350+0000", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213356+0000", 2024-09-11T05:51:31.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3759999999999999e-06 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213372+0000", 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6903000000000001e-05 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.213386+0000", 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3594999999999999e-05 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19273 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.215988+0000", 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00047501299999999999, 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001042956, 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19273 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.215988+0000", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.215988+0000", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.215990+0000", 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5399999999999998e-06 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216001+0000", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0791999999999999e-05 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216107+0000", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010558 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216120+0000", 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3696000000000001e-05 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19274 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216145+0000", 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00031736699999999999, 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00092943300000000002, 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19274 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.262 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216145+0000", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216145+0000", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216146+0000", 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7600000000000006e-07 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216152+0000", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9170000000000003e-06 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216165+0000", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2332e-05 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216172+0000", 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7729999999999999e-06 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.264 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19275 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216185+0000", 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.000278056, 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093560599999999996, 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19275 2024-09-11T05:51:31.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216185+0000", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216185+0000", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216185+0000", 2024-09-11T05:51:31.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3e-07 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216189+0000", 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.213e-06 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216199+0000", 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0453e-05 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216206+0000", 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8789999999999997e-06 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19276 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216216+0000", 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00024633499999999998, 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093201299999999998, 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19276 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216216+0000", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216216+0000", 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216217+0000", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1299999999999999e-07 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216220+0000", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1769999999999998e-06 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216230+0000", 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9089999999999995e-06 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216238+0000", 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19277 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216248+0000", 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00021519400000000001, 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093231599999999998, 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.271 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19277 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216248+0000", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216248+0000", 2024-09-11T05:51:31.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216248+0000", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7700000000000004e-07 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216251+0000", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9100000000000001e-06 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216265+0000", 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4039000000000001e-05 2024-09-11T05:51:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216272+0000", 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19236 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.211833+0000", 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0046296100000000001, 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053743879999999999, 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.274 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19236 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211833+0000", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211833+0000", 2024-09-11T05:51:31.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211834+0000", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1030000000000001e-06 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211840+0000", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.463e-06 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211858+0000", 2024-09-11T05:51:31.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7978999999999998e-05 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.211871+0000", 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3602e-05 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.217188+0000", 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053162310000000003 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.277 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19278 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216282+0000", 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.000180483, 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095529800000000004, 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19278 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216282+0000", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216282+0000", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216283+0000", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4300000000000003e-07 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216286+0000", 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0699999999999998e-06 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216296+0000", 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9660000000000008e-06 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216303+0000", 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8029999999999999e-06 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19237 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212079+0000", 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.004383606, 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051952789999999997, 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19237 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212079+0000", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212079+0000", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212081+0000", 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212089+0000", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6060000000000003e-06 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212115+0000", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5548e-05 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212132+0000", 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7016000000000001e-05 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19279 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216316+0000", 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00014727499999999999, 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098641799999999997, 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19279 2024-09-11T05:51:31.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216316+0000", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216316+0000", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216317+0000", 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0529999999999999e-06 2024-09-11T05:51:31.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216322+0000", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.994e-06 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216357+0000", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5253999999999998e-05 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216372+0000", 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5625e-05 2024-09-11T05:51:31.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19238 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212155+0000", 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0043079839999999999, 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051742469999999999, 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19238 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212155+0000", 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212155+0000", 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212156+0000", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.189e-06 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212162+0000", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7899999999999996e-06 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212181+0000", 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9545e-05 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212199+0000", 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7416000000000001e-05 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19280 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216393+0000", 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.9323e-05, 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096222600000000005, 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19280 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216393+0000", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216393+0000", 2024-09-11T05:51:31.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216395+0000", 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.176e-06 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216400+0000", 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7699999999999998e-06 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216423+0000", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2957999999999999e-05 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216437+0000", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3995999999999999e-05 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19239 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212219+0000", 2024-09-11T05:51:31.293 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0042438889999999998, 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051644799999999999, 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19239 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212219+0000", 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212219+0000", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212220+0000", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212226+0000", 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9789999999999998e-06 2024-09-11T05:51:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212244+0000", 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8182999999999999e-05 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212258+0000", 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3672000000000001e-05 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19281 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216456+0000", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.9269999999999999e-06, 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096865700000000005, 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19281 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216456+0000", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216456+0000", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216457+0000", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216463+0000", 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3060000000000003e-06 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216485+0000", 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1277999999999999e-05 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216499+0000", 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3923e-05 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19240 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212347+0000", 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0041156939999999996, 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051154089999999996, 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19240 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.300 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212347+0000", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212347+0000", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212348+0000", 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.37e-06 2024-09-11T05:51:31.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212355+0000", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.866e-06 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212378+0000", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2829000000000001e-05 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212394+0000", 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5665e-05 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.302 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19282 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216517+0000", 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999456, 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098590500000000003, 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19282 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.303 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216517+0000", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216517+0000", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216518+0000", 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2929999999999999e-06 2024-09-11T05:51:31.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216524+0000", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5600000000000001e-06 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216543+0000", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8972000000000001e-05 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216569+0000", 2024-09-11T05:51:31.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6658000000000001e-05 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19241 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.212416+0000", 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0040471279999999997, 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051166220000000004, 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.306 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19241 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212416+0000", 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212416+0000", 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212417+0000", 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.336e-06 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212423+0000", 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3799999999999999e-06 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212442+0000", 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8892000000000001e-05 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.212459+0000", 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19283 3.1e 3.a533ca9e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.216592+0000", 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998703, 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096797300000000001, 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.309 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19283 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216592+0000", 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216592+0000", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216593+0000", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1039999999999999e-06 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216599+0000", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216624+0000", 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5457999999999999e-05 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.216639+0000", 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4151999999999999e-05 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 30 2024-09-11T05:51:31.312 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:31.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2166 write 0x40731ab thru 0x407f23c (0xc092 bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2167 read 0x1ec8fff thru 0x1ed2f90 (0x9f92 bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2168 trunc from 0x407f23d to 0xb5726a6 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2169 read 0x6d2ac1 thru 0x6e2868 (0xfda8 bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2171 read 0xa73f291 thru 0xa740fcb (0x1d3b bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2173 read 0x18e2004 thru 0x18e2d1e (0xd1b bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2175 punch from 0x64135bb to 0x641d4b2, (0x9ef7 bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2176 read 0x3e02a17 thru 0x3e079ce (0x4fb8 bytes) 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2177 trunc from 0xb5726a6 to 0xa70a993 2024-09-11T05:51:31.315 INFO:teuthology.orchestra.run.smithi165.stdout:2178 writesame 0x5ed6762 thru 0x5edea0f data_size 0x2b(0x82ae bytes) 2024-09-11T05:51:31.316 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:31.313+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:31.329 INFO:teuthology.orchestra.run.smithi165.stdout:2179 write 0x365403f thru 0x36617cd (0xd78f bytes) 2024-09-11T05:51:31.364 INFO:teuthology.orchestra.run.smithi165.stdout:2180 writesame 0xb3e853a thru 0xb3f1d5e data_size 0x1(0x9825 bytes) 2024-09-11T05:51:31.367 INFO:teuthology.orchestra.run.smithi165.stdout:2182 trunc from 0xb3f1d5f to 0x9d8bc4e 2024-09-11T05:51:31.377 INFO:teuthology.orchestra.run.smithi165.stdout:2183 writesame 0x5cb23d5 thru 0x5cc0f76 data_size 0x1(0xeba2 bytes) 2024-09-11T05:51:31.406 INFO:teuthology.orchestra.run.smithi165.stdout:2186 read 0x6ad82b5 thru 0x6adb118 (0x2e64 bytes) 2024-09-11T05:51:31.407 INFO:teuthology.orchestra.run.smithi165.stdout:2187 trunc from 0x9d8bc4e to 0xad4c146 2024-09-11T05:51:31.415 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:31.414+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:31.429 INFO:teuthology.orchestra.run.smithi165.stdout:2189 writesame 0xc0c4628 thru 0xc0d3703 data_size 0xa(0xf0dc bytes) 2024-09-11T05:51:31.429 INFO:teuthology.orchestra.run.smithi165.stdout:2190 read 0xb197c46 thru 0xb1a3cec (0xc0a7 bytes) 2024-09-11T05:51:31.429 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:31.445 INFO:teuthology.orchestra.run.smithi165.stdout:2193 write 0xe71d6d9 thru 0xe729021 (0xb949 bytes) 2024-09-11T05:51:31.466 INFO:teuthology.orchestra.run.smithi165.stdout:2194 read 0x9ed610d thru 0x9edade9 (0x4cdd bytes) 2024-09-11T05:51:31.466 INFO:teuthology.orchestra.run.smithi165.stdout:2195 clone 13 order 24 su 65536 sc 12 2024-09-11T05:51:31.471 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:31.471 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:31.471 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:31.471 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:31.471 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:31.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:31.527 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:31.525+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:31.616 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:31.613+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:31.650 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:31.650 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:31.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:31.650 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:31.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:1198 3.8 3:15f20ad5:::rbd_data.112475afb12f.00000000000000b6:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.499073233000001, 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013528144000000001, 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1198 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149900+0000", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149901+0000", 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.172e-06 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149915+0000", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3801000000000001e-05 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149929+0000", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3624e-05 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149935+0000", 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-11T05:51:31.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.149998+0000", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3065999999999996e-05 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.150144+0000", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014578400000000001 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.152779+0000", 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002635025 2024-09-11T05:51:31.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163265+0000", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010485758 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163327+0000", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2415000000000005e-05 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163372+0000", 2024-09-11T05:51:31.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4331e-05 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:13.163428+0000", 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.664e-05 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:6095 3.b 3:d107d222:::rbd_data.11249b359231.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.286492665000001, 2024-09-11T05:51:31.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012675989, 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 6095 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362481+0000", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362482+0000", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.099e-06 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362488+0000", 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4869999999999999e-06 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362502+0000", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3407999999999999e-05 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362510+0000", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1799999999999996e-06 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362567+0000", 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7538999999999999e-05 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.362653+0000", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6051999999999995e-05 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.364892+0000", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022386630000000001 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.374988+0000", 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010095860999999999 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375041+0000", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2785999999999999e-05 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375105+0000", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4715999999999995e-05 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.375157+0000", 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1198e-05 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.661 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14841 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.017737533, 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013476685, 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14841 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.662 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631236+0000", 2024-09-11T05:51:31.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6600000000000002e-07 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631238+0000", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9120000000000001e-06 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631244+0000", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2809999999999997e-06 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631249+0000", 2024-09-11T05:51:31.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6550000000000003e-06 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640915+0000", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096659480000000006 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.640996+0000", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1108999999999999e-05 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644712+0000", 2024-09-11T05:51:31.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037164139999999999 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14842 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0176982839999997, 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013741382999999999, 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.666 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14842 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631275+0000", 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631276+0000", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631280+0000", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9470000000000002e-06 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631287+0000", 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0450000000000003e-06 2024-09-11T05:51:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631293+0000", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.006e-06 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644746+0000", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013453074000000001 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.644876+0000", 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012993999999999999 2024-09-11T05:51:31.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645016+0000", 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014087299999999999 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14843 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0176732240000002, 2024-09-11T05:51:31.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014071284999999999, 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14843 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631300+0000", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631301+0000", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2600000000000002e-07 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631303+0000", 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.962e-06 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631309+0000", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0650000000000004e-06 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631313+0000", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8380000000000001e-06 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645202+0000", 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013888282 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645283+0000", 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0793999999999999e-05 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645371+0000", 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8818000000000002e-05 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14863 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.674 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0176532119999999, 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014249672, 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14863 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631320+0000", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2e-07 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631322+0000", 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.968e-06 2024-09-11T05:51:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631331+0000", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0949999999999996e-06 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631336+0000", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0969999999999996e-06 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645401+0000", 2024-09-11T05:51:31.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014064933 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645484+0000", 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3325999999999995e-05 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645570+0000", 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5833000000000003e-05 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14864 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4194304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0176308770000002, 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014417572, 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14864 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631342+0000", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631343+0000", 2024-09-11T05:51:31.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6699999999999999e-07 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631365+0000", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2541e-05 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631373+0000", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2989999999999999e-06 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631378+0000", 2024-09-11T05:51:31.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645599+0000", 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01422167 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645674+0000", 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4779999999999999e-05 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645760+0000", 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5867999999999998e-05 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14865 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0175854710000003, 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014562083, 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.683 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14865 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631388+0000", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0500000000000004e-07 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631390+0000", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1090000000000001e-06 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631398+0000", 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0829999999999998e-06 2024-09-11T05:51:31.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631403+0000", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1680000000000003e-06 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645789+0000", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014386144 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645863+0000", 2024-09-11T05:51:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3924000000000006e-05 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645950+0000", 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7150000000000004e-05 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14866 3.b 3:d1985b3d:::rbd_data.1124ef904871.0000000000000001:a [sparse-read 4980736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0175639790000002, 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014728692, 2024-09-11T05:51:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14866 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631409+0000", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631410+0000", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.27e-07 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631412+0000", 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1059999999999998e-06 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631420+0000", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631425+0000", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9649999999999999e-06 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.645979+0000", 2024-09-11T05:51:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014554015999999999 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646051+0000", 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2631000000000003e-05 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646138+0000", 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6828000000000003e-05 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14876 3.b 3:d11944fd:::rbd_data.1124ef904871.0000000000000005:a [sparse-read 3407872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0175420930000003, 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014895617, 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14876 2024-09-11T05:51:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631431+0000", 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631432+0000", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.77e-07 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631434+0000", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9290000000000001e-06 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631440+0000", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2670000000000003e-06 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631445+0000", 2024-09-11T05:51:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7389999999999999e-06 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646167+0000", 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01472234 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646240+0000", 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3596000000000005e-05 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646327+0000", 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6368999999999999e-05 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:14877 3.b 3:d7eb8399:::rbd_data.1124ef904871.000000000000000a:a [sparse-read 6029312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.0175207669999997, 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015071222, 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 14877 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631453+0000", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1399999999999997e-07 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631455+0000", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1619999999999998e-06 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631462+0000", 2024-09-11T05:51:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1810000000000004e-06 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.631476+0000", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3263e-05 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646359+0000", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014883601 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646435+0000", 2024-09-11T05:51:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6056999999999994e-05 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:25.646524+0000", 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8543999999999998e-05 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:15330 3.7 3:e3450b35:::rbd_data.11243bbd48bb.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4494368790000003, 2024-09-11T05:51:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018829538, 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 15330 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199536+0000", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199537+0000", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-07 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199542+0000", 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6970000000000001e-06 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199553+0000", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0831e-05 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199560+0000", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099999999999997e-06 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199598+0000", 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7697999999999997e-05 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.199639+0000", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1863000000000002e-05 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.202516+0000", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028767580000000001 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218204+0000", 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015687326000000001 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218230+0000", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6387e-05 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218316+0000", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6227e-05 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.218366+0000", 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9795e-05 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19028 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 753664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.085551+0000", 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.56342211799999997, 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011978608, 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19028 2024-09-11T05:51:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085551+0000", 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085551+0000", 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085553+0000", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.35e-06 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085559+0000", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7499999999999997e-06 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085580+0000", 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1055000000000001e-05 2024-09-11T05:51:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.085595+0000", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4525000000000001e-05 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.097319+0000", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011724313 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.097403+0000", 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3866e-05 2024-09-11T05:51:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.097530+0000", 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126749 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19138 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 376832~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096156+0000", 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55281717500000005, 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012127275, 2024-09-11T05:51:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19138 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096156+0000", 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096156+0000", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096157+0000", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096166+0000", 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:51:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096188+0000", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1798e-05 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096202+0000", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4609e-05 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.107877+0000", 2024-09-11T05:51:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011674812999999999 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108086+0000", 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209015 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108283+0000", 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019736999999999999 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19140 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 540672~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096223+0000", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55275038600000004, 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012596754999999999, 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19140 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096223+0000", 2024-09-11T05:51:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096223+0000", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096224+0000", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699999999999999e-06 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096230+0000", 2024-09-11T05:51:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-06 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096249+0000", 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8601999999999999e-05 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096263+0000", 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4439e-05 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108379+0000", 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012116408 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108570+0000", 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019089500000000001 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108820+0000", 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024937100000000002 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19141 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 1015808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096282+0000", 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55269083600000002, 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012825543999999999, 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19141 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096282+0000", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096282+0000", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096284+0000", 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096289+0000", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7259999999999999e-06 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096308+0000", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8831999999999999e-05 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096322+0000", 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3362e-05 2024-09-11T05:51:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108890+0000", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012568317000000001 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.108980+0000", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9803999999999996e-05 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109108+0000", 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012837300000000001 2024-09-11T05:51:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19142 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 704512~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096343+0000", 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55263034700000002, 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012995072, 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19142 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096343+0000", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096343+0000", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096344+0000", 2024-09-11T05:51:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.15e-06 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096349+0000", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3700000000000003e-06 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096368+0000", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8754999999999999e-05 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096383+0000", 2024-09-11T05:51:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4626e-05 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109143+0000", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012759968 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109221+0000", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8145000000000005e-05 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109338+0000", 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117058 2024-09-11T05:51:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19150 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096401+0000", 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55257191100000003, 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013141867, 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19150 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096401+0000", 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096401+0000", 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096403+0000", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.147e-06 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096416+0000", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3089e-05 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096434+0000", 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8842e-05 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096448+0000", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3638999999999999e-05 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109370+0000", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012922289 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109450+0000", 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9417000000000002e-05 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109543+0000", 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3443999999999995e-05 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19157 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 966656~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096466+0000", 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55250685799999999, 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013289779999999999, 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19157 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096466+0000", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096466+0000", 2024-09-11T05:51:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096468+0000", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096473+0000", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3859999999999998e-06 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096491+0000", 2024-09-11T05:51:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8204999999999999e-05 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096505+0000", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3529999999999999e-05 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109573+0000", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013068207 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109650+0000", 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7615999999999997e-05 2024-09-11T05:51:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.109756+0000", 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105723 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:19174 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 999424~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.096522+0000", 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.55245084899999997, 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01379289, 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 19174 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096522+0000", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096522+0000", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096524+0000", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4470000000000001e-06 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096529+0000", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1329999999999998e-06 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096547+0000", 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8059000000000001e-05 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.096561+0000", 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3889e-05 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.110013+0000", 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013451849 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.110165+0000", 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015239299999999999 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.110315+0000", 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015012000000000001 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:31.739 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:31.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:51:31.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:31.742 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:31.741+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:31.842 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:31.837+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:31.875 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:31.875 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:31.875 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:31.875 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:31.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:31.930 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone12 from 0xe729022 (overlap 0x30570fb) to 0x1356064 2024-09-11T05:51:31.944 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:31.941+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:32.030 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:32.030 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:32.030 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:32.031 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:32.031 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:32.044 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:32.045+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:32.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:32.145 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:32.145+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:32.215 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11636 3.1a 3:5ee2e31d:::rbd_data.1124518b2c12.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.997495892, 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023307497, 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11636 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217167+0000", 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0000000000000001e-06 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221731+0000", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045638420000000002 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221766+0000", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221782+0000", 2024-09-11T05:51:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5899000000000001e-05 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221933+0000", 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015070599999999999 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.222312+0000", 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037949700000000002 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.224398+0000", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00208618 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.237521+0000", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013122642 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240180+0000", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002659509 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240337+0000", 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240471+0000", 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013454299999999999 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12094 3.14 3:29cbd9d3:::rbd_data.1124518b2c12.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 3104699~59461] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.491066171, 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021958854, 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12094 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723593+0000", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.723596+0000", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0800000000000002e-06 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728015+0000", 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004418684 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728048+0000", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3299000000000001e-05 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728064+0000", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5583e-05 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728187+0000", 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122583 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.728429+0000", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024270499999999999 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.729310+0000", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088068599999999999 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.743989+0000", 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01467861 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745369+0000", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001379983 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745475+0000", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010624900000000001 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.745552+0000", 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7391999999999994e-05 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19075 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 589824~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.087910+0000", 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1267498890000001, 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020214612, 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19075 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087910+0000", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087910+0000", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087910+0000", 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:51:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087913+0000", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.106e-06 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087922+0000", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4759999999999998e-06 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087929+0000", 2024-09-11T05:51:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8059999999999997e-06 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.107604+0000", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019675059000000002 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.107837+0000", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023315900000000001 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108124+0000", 2024-09-11T05:51:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028738599999999999 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19076 3.9 3:923190e7:::rbd_data.1124cb371d7e.000000000000004c:head [read 0~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.087937+0000", 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.126722199, 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020468672, 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19076 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087937+0000", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087937+0000", 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087938+0000", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2399999999999998e-07 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087941+0000", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.909e-06 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087949+0000", 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5720000000000003e-06 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087957+0000", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0929999999999997e-06 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108160+0000", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020203559999999999 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108261+0000", 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101266 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108406+0000", 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014464799999999999 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19077 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 753664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.087965+0000", 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1266943519999999, 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021060395999999999, 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19077 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087965+0000", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087965+0000", 2024-09-11T05:51:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087966+0000", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4000000000000001e-07 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087969+0000", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087978+0000", 2024-09-11T05:51:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7520000000000002e-06 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.087985+0000", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.063e-06 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108499+0000", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020514251000000001 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.108738+0000", 2024-09-11T05:51:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023959400000000001 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109026+0000", 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000287163 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19082 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 131072~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.088239+0000", 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1264207580000001, 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021091948999999999, 2024-09-11T05:51:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19082 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088239+0000", 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088239+0000", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088240+0000", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5300000000000002e-07 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088244+0000", 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3069999999999996e-06 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088255+0000", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1282e-05 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088266+0000", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0628999999999999e-05 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109060+0000", 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020793646999999998 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109177+0000", 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117244 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109331+0000", 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015388700000000001 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19090 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 294912~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.088846+0000", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1258139359999999, 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021110706999999999, 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19090 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088846+0000", 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088846+0000", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088847+0000", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9699999999999994e-07 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088852+0000", 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.366e-06 2024-09-11T05:51:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088866+0000", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4270999999999999e-05 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088876+0000", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2590000000000006e-06 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109424+0000", 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020548381000000001 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109667+0000", 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024324700000000001 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.109956+0000", 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028918599999999998 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19094 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 114688~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.088961+0000", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1256985720000001, 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021347952, 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19094 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088961+0000", 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088961+0000", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088962+0000", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088966+0000", 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3989999999999997e-06 2024-09-11T05:51:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088978+0000", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1596e-05 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.088986+0000", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1720000000000007e-06 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110034+0000", 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021047960000000001 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110155+0000", 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121147 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110309+0000", 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153764 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19095 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 458752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.089070+0000", 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1255893699999999, 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021862677, 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19095 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089070+0000", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089070+0000", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089071+0000", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0899999999999999e-06 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089076+0000", 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0300000000000001e-06 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089090+0000", 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3351e-05 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089099+0000", 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1929999999999999e-06 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110400+0000", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021300942999999999 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110650+0000", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025000599999999999 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.110933+0000", 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028306400000000003 2024-09-11T05:51:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19096 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 622592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.089316+0000", 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1253439059999999, 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022235979999999999, 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19096 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089316+0000", 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089316+0000", 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089317+0000", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7599999999999996e-07 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089321+0000", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.437e-06 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089334+0000", 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3434999999999999e-05 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089344+0000", 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1090000000000004e-06 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111021+0000", 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021677512999999999 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111270+0000", 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024863000000000001 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111552+0000", 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028197999999999998 2024-09-11T05:51:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19097 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 278528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1250874200000001, 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022586779000000001, 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19097 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089573+0000", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1370000000000001e-06 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089579+0000", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3249999999999998e-06 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089593+0000", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4175999999999999e-05 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089602+0000", 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9320000000000002e-06 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111635+0000", 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022033437999999999 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111872+0000", 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023689199999999999 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112159+0000", 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028687899999999998 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19098 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 442368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.124364261, 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022463285999999999, 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19098 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090297+0000", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090304+0000", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2830000000000004e-06 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090327+0000", 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3264e-05 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090338+0000", 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0285000000000001e-05 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112249+0000", 2024-09-11T05:51:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021911336 2024-09-11T05:51:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112482+0000", 2024-09-11T05:51:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233005 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112759+0000", 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000276483 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19099 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 606208~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1241589590000001, 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022863720000000001, 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19099 2024-09-11T05:51:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090502+0000", 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1829999999999999e-06 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090509+0000", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9229999999999996e-06 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090525+0000", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6551000000000001e-05 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090535+0000", 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0071999999999999e-05 2024-09-11T05:51:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112846+0000", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022310399000000002 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113083+0000", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000237445 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113364+0000", 2024-09-11T05:51:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028114699999999998 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19100 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1241078449999999, 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023303653000000001, 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19100 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4700000000000001e-07 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090556+0000", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4429999999999999e-06 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090565+0000", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.302e-06 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090572+0000", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.233e-06 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113495+0000", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022922464 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113656+0000", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016068899999999999 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113855+0000", 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019987499999999999 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19105 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.124073992, 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023901565999999999, 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19105 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5300000000000004e-07 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090589+0000", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.19e-06 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090598+0000", 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8659999999999995e-06 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090605+0000", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113961+0000", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023356247 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.114202+0000", 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000240261 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.114487+0000", 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028545300000000002 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19117 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 720896~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.096356+0000", 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1183037140000001, 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020255875, 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19117 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096356+0000", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096356+0000", 2024-09-11T05:51:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096357+0000", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1299999999999999e-07 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096360+0000", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096369+0000", 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.789e-06 2024-09-11T05:51:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096376+0000", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5460000000000003e-06 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116338+0000", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019962079000000001 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116463+0000", 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124084 2024-09-11T05:51:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116612+0000", 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149214 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19137 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 917504~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.096385+0000", 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.118274277, 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02057734, 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19137 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096385+0000", 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096385+0000", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096386+0000", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7700000000000004e-07 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096389+0000", 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.185e-06 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096398+0000", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0759999999999992e-06 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096406+0000", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4499999999999998e-06 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116677+0000", 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020271768999999999 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116802+0000", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012473399999999999 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.116963+0000", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016044900000000001 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19146 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 999424~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.096415+0000", 2024-09-11T05:51:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.118244907, 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020891554, 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19146 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096415+0000", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096415+0000", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096415+0000", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8299999999999996e-07 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096419+0000", 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1860000000000001e-06 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096427+0000", 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7690000000000002e-06 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096435+0000", 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3860000000000001e-06 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117026+0000", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020591363000000001 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117147+0000", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0001205 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117306+0000", 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159667 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19154 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.096449+0000", 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1182105609999999, 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021194648, 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19154 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096449+0000", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096449+0000", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096450+0000", 2024-09-11T05:51:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5000000000000002e-07 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096453+0000", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096462+0000", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1959999999999997e-06 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096470+0000", 2024-09-11T05:51:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8189999999999992e-06 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117368+0000", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020898068999999998 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117489+0000", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120877 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117644+0000", 2024-09-11T05:51:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015499700000000001 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19172 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 901120~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.096479+0000", 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1181801170000001, 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021490636, 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19172 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096479+0000", 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096479+0000", 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096480+0000", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1299999999999999e-07 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096485+0000", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6990000000000002e-06 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096494+0000", 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8130000000000002e-06 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.096501+0000", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2819999999999999e-06 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117695+0000", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021194265 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117813+0000", 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118251 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.117970+0000", 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156613 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.305 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:32.306 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-09-11T05:51:32.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:32.307 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:32.305+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:32.407 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:32.405+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:32.477 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:32.477 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:32.477 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:32.478 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:32.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:32.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:32.498 INFO:teuthology.orchestra.run.smithi165.stdout:2196 trunc from 0xe729022 to 0xc89939a 2024-09-11T05:51:32.508 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:32.505+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:32.547 INFO:teuthology.orchestra.run.smithi165.stdout:2197 trunc from 0xc89939a to 0x8a14895 2024-09-11T05:51:32.614 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:32.605+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:32.646 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:32.646 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:32.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:32.715 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:32.709+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:32.720 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:51:32.736 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:32.736 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:32.775 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:32.775 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:32.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.051561750999999, 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.810 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:32.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12821 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8301934600000003, 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028055824, 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12821 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979066+0000", 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8839999999999998e-06 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979079+0000", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3159999999999999e-05 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979111+0000", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2406000000000001e-05 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979128+0000", 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6733000000000001e-05 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979265+0000", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013638600000000001 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979400+0000", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135752 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.983348+0000", 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039480169999999998 2024-09-11T05:51:32.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.006806+0000", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023457281999999999 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.006917+0000", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111462 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.007011+0000", 2024-09-11T05:51:32.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3968999999999994e-05 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.007119+0000", 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107773 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21842 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.598967+0000", 2024-09-11T05:51:32.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.21028941300000001, 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018247289, 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21842 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.598967+0000", 2024-09-11T05:51:32.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.598967+0000", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.598968+0000", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.64e-07 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.598972+0000", 2024-09-11T05:51:32.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6950000000000002e-06 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.598985+0000", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2863e-05 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.599005+0000", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0882000000000001e-05 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.616914+0000", 2024-09-11T05:51:32.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017909021000000001 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617044+0000", 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012960900000000001 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617214+0000", 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017025499999999999 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21843 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 442368~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.599018+0000", 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.21023862300000001, 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018528950999999998, 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21843 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.599018+0000", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.599018+0000", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.599019+0000", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1999999999999998e-07 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.599023+0000", 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8020000000000002e-06 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600128+0000", 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001105574 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600135+0000", 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0380000000000002e-06 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617266+0000", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017130551000000001 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617407+0000", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141118 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617547+0000", 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013994800000000001 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21845 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600308+0000", 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20894858499999999, 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018229840000000001, 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21845 2024-09-11T05:51:32.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600308+0000", 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600308+0000", 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600309+0000", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7299999999999995e-07 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600312+0000", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.765e-06 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600319+0000", 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600325+0000", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5049999999999998e-06 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.617918+0000", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017593761999999999 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618046+0000", 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127415 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618538+0000", 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049179400000000004 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21847 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600332+0000", 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208923951, 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018421679999999999, 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21847 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600332+0000", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600332+0000", 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600333+0000", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6400000000000003e-07 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600335+0000", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4030000000000001e-06 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600341+0000", 2024-09-11T05:51:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.083e-06 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600347+0000", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0880000000000002e-06 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618596+0000", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018249656999999999 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618681+0000", 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4500999999999996e-05 2024-09-11T05:51:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618754+0000", 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3484000000000003e-05 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21848 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 868352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208901796, 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025560946000000001, 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21848 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4400000000000001e-07 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600357+0000", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9300000000000002e-06 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600363+0000", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4489999999999996e-06 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600368+0000", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.977e-06 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618805+0000", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018436113000000001 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618884+0000", 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9640000000000003e-05 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.625916+0000", 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070313930000000004 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21850 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600375+0000", 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20888158800000001, 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018766317000000001, 2024-09-11T05:51:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21850 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600375+0000", 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600375+0000", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600375+0000", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2300000000000002e-07 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600377+0000", 2024-09-11T05:51:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999998e-06 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600384+0000", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3980000000000004e-06 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600390+0000", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0530000000000003e-06 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618977+0000", 2024-09-11T05:51:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018587151 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619073+0000", 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5849999999999999e-05 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619141+0000", 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8342000000000004e-05 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21852 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600398+0000", 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20885877999999999, 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018924764, 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21852 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600398+0000", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600398+0000", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600398+0000", 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2100000000000002e-07 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600406+0000", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600413+0000", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1079999999999998e-06 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600418+0000", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3789999999999997e-06 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619168+0000", 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01874984 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619254+0000", 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6631000000000005e-05 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619322+0000", 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7974999999999999e-05 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21878 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 8192~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600425+0000", 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20883104799999999, 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019097100999999998, 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21878 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600425+0000", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600425+0000", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600426+0000", 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.32e-07 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600428+0000", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0810000000000001e-06 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600434+0000", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2790000000000004e-06 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600439+0000", 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7849999999999999e-06 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619350+0000", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018911196000000002 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619434+0000", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4264000000000005e-05 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619522+0000", 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8064000000000003e-05 2024-09-11T05:51:32.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21879 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 81920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600445+0000", 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20881129300000001, 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019267345000000002, 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21879 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600445+0000", 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600445+0000", 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600446+0000", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600448+0000", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.024e-06 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600454+0000", 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1349999999999997e-06 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600459+0000", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5600000000000001e-06 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619550+0000", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019091172999999999 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619626+0000", 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5727000000000002e-05 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619712+0000", 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6335999999999995e-05 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21880 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 155648~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600472+0000", 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208784094, 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019429776999999999, 2024-09-11T05:51:32.859 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21880 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600472+0000", 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600472+0000", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600473+0000", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600475+0000", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4140000000000001e-06 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600587+0000", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111832 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600593+0000", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1820000000000003e-06 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619739+0000", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019146136000000001 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619814+0000", 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5115000000000002e-05 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619902+0000", 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7678999999999999e-05 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21881 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 294912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600603+0000", 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20865310200000001, 2024-09-11T05:51:32.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019484722999999999, 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21881 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600603+0000", 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600603+0000", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600604+0000", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3499999999999995e-07 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600607+0000", 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-06 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600614+0000", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6259999999999997e-06 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600619+0000", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3369999999999999e-06 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.619928+0000", 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019308868999999999 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620002+0000", 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4234999999999997e-05 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620088+0000", 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5813e-05 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.867 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21882 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 368640~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600626+0000", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208630285, 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019648572, 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21882 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600626+0000", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600626+0000", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600627+0000", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5299999999999999e-07 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600629+0000", 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3080000000000002e-06 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600637+0000", 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1850000000000004e-06 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600643+0000", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6300000000000003e-06 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620116+0000", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019472931999999998 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620192+0000", 2024-09-11T05:51:32.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6542e-05 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620275+0000", 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2521999999999994e-05 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21883 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 442368~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600650+0000", 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208606861, 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019806735999999998, 2024-09-11T05:51:32.872 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21883 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600650+0000", 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600650+0000", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600650+0000", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4299999999999998e-07 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600652+0000", 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2840000000000001e-06 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600659+0000", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.336e-06 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600663+0000", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7129999999999996e-06 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620300+0000", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019637079000000002 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620374+0000", 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3598999999999998e-05 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620456+0000", 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2281999999999996e-05 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21884 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 581632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600670+0000", 2024-09-11T05:51:32.876 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20858658899999999, 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019958231, 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21884 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600670+0000", 2024-09-11T05:51:32.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600670+0000", 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600670+0000", 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4799999999999999e-07 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600672+0000", 2024-09-11T05:51:32.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.826e-06 2024-09-11T05:51:32.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600678+0000", 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600685+0000", 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6479999999999997e-06 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620484+0000", 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019798517000000002 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620560+0000", 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6693000000000006e-05 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620628+0000", 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7749000000000005e-05 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21885 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 655360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600692+0000", 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20856402700000001, 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020122608, 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21885 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600692+0000", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600692+0000", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600693+0000", 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9900000000000001e-07 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600695+0000", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8679999999999999e-06 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600702+0000", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.917e-06 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600707+0000", 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2379999999999997e-06 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620656+0000", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019949132000000001 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620741+0000", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4872999999999999e-05 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620815+0000", 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4181e-05 2024-09-11T05:51:32.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21886 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600714+0000", 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.20854229599999999, 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020299128999999999, 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21886 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600714+0000", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600714+0000", 2024-09-11T05:51:32.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600715+0000", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2399999999999999e-07 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600717+0000", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0109999999999999e-06 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600723+0000", 2024-09-11T05:51:32.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3099999999999997e-06 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600728+0000", 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6829999999999999e-06 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620840+0000", 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020112577 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620945+0000", 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104845 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621013+0000", 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8279000000000001e-05 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21887 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 868352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600733+0000", 2024-09-11T05:51:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208522921, 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020441912999999999, 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21887 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600733+0000", 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600733+0000", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600734+0000", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600737+0000", 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.757e-06 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600743+0000", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4239999999999998e-06 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600748+0000", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7960000000000003e-06 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621039+0000", 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020291192999999999 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621113+0000", 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4036999999999996e-05 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621175+0000", 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2255999999999995e-05 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21888 3.5 3:a01cb852:::rbd_data.11244d01bf21.0000000000000026:f [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:32.895 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600761+0000", 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.208495615, 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020580016999999999, 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21888 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:32.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600761+0000", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600761+0000", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600761+0000", 2024-09-11T05:51:32.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.32e-07 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600764+0000", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1e-06 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600911+0000", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146236 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600918+0000", 2024-09-11T05:51:32.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4279999999999999e-06 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621201+0000", 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020282960999999999 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621278+0000", 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7457000000000001e-05 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:32.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.621341+0000", 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2403000000000005e-05 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:32.900 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:32.901 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:32.901 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:32.901 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:32.902 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:32.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:32.902 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:32.903 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:32.903 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:32.903 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:32.903 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:32.904 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:32.904 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:32.904 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:32.897+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:33.002 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:33.001+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:33.062 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.062 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:33.077 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.077 INFO:teuthology.orchestra.run.smithi165.stdout:2202 write 0x7e43df2 thru 0x7e52d02 (0xef11 bytes) 2024-09-11T05:51:33.082 INFO:teuthology.orchestra.run.smithi165.stdout:2204 writesame 0x34627cf thru 0x346cc58 data_size 0x1(0xa48a bytes) 2024-09-11T05:51:33.103 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:33.101+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:33.125 INFO:teuthology.orchestra.run.smithi165.stdout:2205 flatten 2024-09-11T05:51:33.159 INFO:teuthology.orchestra.run.smithi165.stdout:2207 trunc from 0x8a14895 to 0x24e71e 2024-09-11T05:51:33.203 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:33.201+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:33.224 INFO:teuthology.orchestra.run.smithi165.stdout:2208 write 0xc9f7958 thru 0xc9fe1bc (0x6865 bytes) 2024-09-11T05:51:33.232 INFO:teuthology.orchestra.run.smithi165.stdout:2209 read 0x5ee833c thru 0x5eecb3f (0x4804 bytes) 2024-09-11T05:51:33.232 INFO:teuthology.orchestra.run.smithi165.stdout:2210 write 0xba8994 thru 0xbabb9f (0x320c bytes) 2024-09-11T05:51:33.236 INFO:teuthology.orchestra.run.smithi165.stdout:2212 write 0x462c59d thru 0x463c272 (0xfcd6 bytes) 2024-09-11T05:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:33.258 INFO:teuthology.orchestra.run.smithi165.stdout:2213 write 0xde904c8 thru 0xde97a4d (0x7586 bytes) 2024-09-11T05:51:33.266 INFO:teuthology.orchestra.run.smithi165.stdout:2214 punch from 0x8fb02b3 to 0x8fbac17, (0xa964 bytes) 2024-09-11T05:51:33.266 INFO:teuthology.orchestra.run.smithi165.stdout:2215 read 0x82dc3ee thru 0x82e46d3 (0x82e6 bytes) 2024-09-11T05:51:33.267 INFO:teuthology.orchestra.run.smithi165.stdout:2217 writesame 0x2a9e0f9 thru 0x2aac4b1 data_size 0x1(0xe3b9 bytes) 2024-09-11T05:51:33.305 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:33.301+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:33.338 INFO:teuthology.orchestra.run.smithi165.stdout:2218 read 0xaa64951 thru 0xaa6ed45 (0xa3f5 bytes) 2024-09-11T05:51:33.339 INFO:teuthology.orchestra.run.smithi165.stdout:2219 write 0x2cdeb93 thru 0x2ce7c5f (0x90cd bytes) 2024-09-11T05:51:33.344 INFO:teuthology.orchestra.run.smithi165.stdout:2220 punch from 0xddb37c1 to 0xddbaa15, (0x7254 bytes) 2024-09-11T05:51:33.344 INFO:teuthology.orchestra.run.smithi165.stdout:2221 writesame 0x83def3e thru 0x83e1834 data_size 0x1(0x28f7 bytes) 2024-09-11T05:51:33.366 INFO:teuthology.orchestra.run.smithi165.stdout:2222 trunc from 0xde97a4e to 0xa06ad92 2024-09-11T05:51:33.382 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.382 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:33.405 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:33.401+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:33.411 INFO:teuthology.orchestra.run.smithi165.stdout:2223 writesame 0xa228346 thru 0xa228587 data_size 0x2(0x242 bytes) 2024-09-11T05:51:33.413 INFO:teuthology.orchestra.run.smithi165.stdout:2224 read 0x6482523 thru 0x648f192 (0xcc70 bytes) 2024-09-11T05:51:33.415 INFO:teuthology.orchestra.run.smithi165.stdout:2225 punch from 0x4be64d6 to 0x4beef37, (0x8a61 bytes) 2024-09-11T05:51:33.415 INFO:teuthology.orchestra.run.smithi165.stdout:2226 write 0x40603b6 thru 0x406c8f2 (0xc53d bytes) 2024-09-11T05:51:33.419 INFO:teuthology.orchestra.run.smithi165.stdout:2227 punch from 0x1cabd7a to 0x1cbac41, (0xeec7 bytes) 2024-09-11T05:51:33.419 INFO:teuthology.orchestra.run.smithi165.stdout:2228 write 0x1cf7582 thru 0x1d0305f (0xbade bytes) 2024-09-11T05:51:33.422 INFO:teuthology.orchestra.run.smithi165.stdout:2229 read 0x97f53eb thru 0x98005c6 (0xb1dc bytes) 2024-09-11T05:51:33.423 INFO:teuthology.orchestra.run.smithi165.stdout:2230 read 0x96db8b3 thru 0x96dc2dc (0xa2a bytes) 2024-09-11T05:51:33.424 INFO:teuthology.orchestra.run.smithi165.stdout:2232 writesame 0x3f5ea37 thru 0x3f5f382 data_size 0x22(0x94c bytes) 2024-09-11T05:51:33.426 INFO:teuthology.orchestra.run.smithi165.stdout:2233 punch from 0x3a308c0 to 0x3a35db6, (0x54f6 bytes) 2024-09-11T05:51:33.450 INFO:teuthology.orchestra.run.smithi165.stdout:2234 write 0xd93e5fd thru 0xd943a7d (0x5481 bytes) 2024-09-11T05:51:33.459 INFO:teuthology.orchestra.run.smithi165.stdout:2235 writesame 0x13c22b9 thru 0x13cbc31 data_size 0x1(0x9979 bytes) 2024-09-11T05:51:33.485 INFO:teuthology.orchestra.run.smithi165.stdout:2236 read 0x4d17fe7 thru 0x4d20cee (0x8d08 bytes) 2024-09-11T05:51:33.486 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.506 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:33.501+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:33.518 INFO:teuthology.orchestra.run.smithi165.stdout:2239 trunc from 0xd943a7e to 0x1d15c37 2024-09-11T05:51:33.525 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.525 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:33.548 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.572 INFO:teuthology.orchestra.run.smithi165.stdout:2242 trunc from 0x1d15c37 to 0xa608793 2024-09-11T05:51:33.584 INFO:teuthology.orchestra.run.smithi165.stdout:2243 write 0x9b8eccb thru 0x9b97e77 (0x91ad bytes) 2024-09-11T05:51:33.598 INFO:teuthology.orchestra.run.smithi165.stdout:2244 writesame 0xa76665d thru 0xa76e4a5 data_size 0x1(0x7e49 bytes) 2024-09-11T05:51:33.600 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.606 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:33.601+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:33.623 INFO:teuthology.orchestra.run.smithi165.stdout:2246 writesame 0xde19f83 thru 0xde27607 data_size 0x1(0xd685 bytes) 2024-09-11T05:51:33.623 INFO:teuthology.orchestra.run.smithi165.stdout:2247 read 0x8f1f4b0 thru 0x8f2cabd (0xd60e bytes) 2024-09-11T05:51:33.623 INFO:teuthology.orchestra.run.smithi165.stdout:2248 punch from 0x4184234 to 0x418eca1, (0xaa6d bytes) 2024-09-11T05:51:33.623 INFO:teuthology.orchestra.run.smithi165.stdout:2249 write 0x3e5cc7 thru 0x3ed747 (0x7a81 bytes) 2024-09-11T05:51:33.626 INFO:teuthology.orchestra.run.smithi165.stdout:2250 read 0x8b394a1 thru 0x8b4383e (0xa39e bytes) 2024-09-11T05:51:33.627 INFO:teuthology.orchestra.run.smithi165.stdout:2253 read 0x2963022 thru 0x29652a1 (0x2280 bytes) 2024-09-11T05:51:33.628 INFO:teuthology.orchestra.run.smithi165.stdout:2254 punch from 0xbbd302a to 0xbbd727b, (0x4251 bytes) 2024-09-11T05:51:33.628 INFO:teuthology.orchestra.run.smithi165.stdout:2259 read 0xddbfe9b thru 0xddc2d96 (0x2efc bytes) 2024-09-11T05:51:33.628 INFO:teuthology.orchestra.run.smithi165.stdout:2260 write 0x18fdf1f thru 0x19025ac (0x468e bytes) 2024-09-11T05:51:33.631 INFO:teuthology.orchestra.run.smithi165.stdout:2263 read 0x6a35598 thru 0x6a3eb29 (0x9592 bytes) 2024-09-11T05:51:33.632 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.632 INFO:teuthology.orchestra.run.smithi165.stdout:2265 write 0xdbe2dde thru 0xdbeb917 (0x8b3a bytes) 2024-09-11T05:51:33.654 INFO:teuthology.orchestra.run.smithi165.stdout:2268 trunc from 0xde27608 to 0x82468ef 2024-09-11T05:51:33.666 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.667 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:33.685 INFO:teuthology.orchestra.run.smithi165.stdout:2269 read 0x8106e9d thru 0x8115b81 (0xece5 bytes) 2024-09-11T05:51:33.701 INFO:teuthology.orchestra.run.smithi165.stdout:2270 writesame 0x962848e thru 0x962f36b data_size 0x17(0x6ede bytes) 2024-09-11T05:51:33.716 INFO:teuthology.orchestra.run.smithi165.stdout:2272 trunc from 0x962f36c to 0xe8c1580 2024-09-11T05:51:33.729 INFO:teuthology.orchestra.run.smithi165.stdout:2273 read 0x724f774 thru 0x725f3e4 (0xfc71 bytes) 2024-09-11T05:51:33.730 INFO:teuthology.orchestra.run.smithi165.stdout:2274 read 0x3401afe thru 0x340fe48 (0xe34b bytes) 2024-09-11T05:51:33.732 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:33.733+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:33.755 INFO:teuthology.orchestra.run.smithi165.stdout:2278 trunc from 0xe8c1580 to 0x486db8a 2024-09-11T05:51:33.794 INFO:teuthology.orchestra.run.smithi165.stdout:2280 write 0x6afc774 thru 0x6afc83c (0xc9 bytes) 2024-09-11T05:51:33.808 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:33.809+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:33.809 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.809 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:33.830 INFO:teuthology.orchestra.run.smithi165.stdout:2286 writesame 0xe734f6d thru 0xe739c2c data_size 0x20(0x4cc0 bytes) 2024-09-11T05:51:33.830 INFO:teuthology.orchestra.run.smithi165.stdout:2287 write 0xf7bf8b thru 0xf85521 (0x9597 bytes) 2024-09-11T05:51:33.834 INFO:teuthology.orchestra.run.smithi165.stdout:2288 punch from 0x14da58b to 0x14dc7db, (0x2250 bytes) 2024-09-11T05:51:33.834 INFO:teuthology.orchestra.run.smithi165.stdout:2289 writesame 0x75e035e thru 0x75e1d38 data_size 0x1(0x19db bytes) 2024-09-11T05:51:33.862 INFO:teuthology.orchestra.run.smithi165.stdout:2290 trunc from 0xe739c2d to 0x2c26c6f 2024-09-11T05:51:33.876 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.876 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.877 INFO:teuthology.orchestra.run.smithi165.stdout:2295 read 0x15929cf thru 0x159dfc8 (0xb5fa bytes) 2024-09-11T05:51:33.877 INFO:teuthology.orchestra.run.smithi165.stdout:2296 punch from 0x5301be to 0x540053, (0xfe95 bytes) 2024-09-11T05:51:33.877 INFO:teuthology.orchestra.run.smithi165.stdout:2298 read 0x3687e thru 0x3a76a (0x3eed bytes) 2024-09-11T05:51:33.877 INFO:teuthology.orchestra.run.smithi165.stdout:2299 read 0x29d2c60 thru 0x29d785e (0x4bff bytes) 2024-09-11T05:51:33.878 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.881 INFO:teuthology.orchestra.run.smithi165.stdout:2302 trunc from 0x2c26c6f to 0x4c169a9 2024-09-11T05:51:33.908 INFO:teuthology.orchestra.run.smithi165.stdout:2305 writesame 0xde7d09a thru 0xde86014 data_size 0x1(0x8f7b bytes) 2024-09-11T05:51:33.908 INFO:teuthology.orchestra.run.smithi165.stdout:2306 write 0xa9a379d thru 0xa9a633a (0x2b9e bytes) 2024-09-11T05:51:33.908 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:33.909+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:33.910 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:33.910 INFO:teuthology.orchestra.run.smithi165.stdout:2309 write 0x58fd4a0 thru 0x5909595 (0xc0f6 bytes) 2024-09-11T05:51:33.913 INFO:teuthology.orchestra.run.smithi165.stdout:2310 write 0x939d0ab thru 0x93a5826 (0x877c bytes) 2024-09-11T05:51:33.915 INFO:teuthology.orchestra.run.smithi165.stdout:2311 write 0x5e63f26 thru 0x5e72100 (0xe1db bytes) 2024-09-11T05:51:33.921 INFO:teuthology.orchestra.run.smithi165.stdout:2312 punch from 0x3c60fe3 to 0x3c65d6f, (0x4d8c bytes) 2024-09-11T05:51:33.921 INFO:teuthology.orchestra.run.smithi165.stdout:2314 write 0xdd96a53 thru 0xdd9cbf3 (0x61a1 bytes) 2024-09-11T05:51:33.927 INFO:teuthology.orchestra.run.smithi165.stdout:2316 read 0xcef829f thru 0xcf08229 (0xff8b bytes) 2024-09-11T05:51:33.927 INFO:teuthology.orchestra.run.smithi165.stdout:2317 write 0x6be8a75 thru 0x6bf21d3 (0x975f bytes) 2024-09-11T05:51:33.932 INFO:teuthology.orchestra.run.smithi165.stdout:2318 writesame 0x18f7335 thru 0x18f845d data_size 0x1(0x1129 bytes) 2024-09-11T05:51:33.938 INFO:teuthology.orchestra.run.smithi165.stdout:2319 write 0x390ef64 thru 0x391e412 (0xf4af bytes) 2024-09-11T05:51:33.941 INFO:teuthology.orchestra.run.smithi165.stdout:2320 read 0x8f0dea8 thru 0x8f1932e (0xb487 bytes) 2024-09-11T05:51:33.947 INFO:teuthology.orchestra.run.smithi165.stdout:2321 trunc from 0xde86015 to 0xa5c3110 2024-09-11T05:51:33.966 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:33.966 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:33.974 INFO:teuthology.orchestra.run.smithi165.stdout:2322 writesame 0x94de160 thru 0x94e707f data_size 0x50(0x8f20 bytes) 2024-09-11T05:51:33.978 INFO:teuthology.orchestra.run.smithi165.stdout:2323 write 0x10b31f7 thru 0x10b5dac (0x2bb6 bytes) 2024-09-11T05:51:33.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:33.987 INFO:teuthology.orchestra.run.smithi165.stdout:2324 trunc from 0xa5c3110 to 0xe2a656f 2024-09-11T05:51:34.002 INFO:teuthology.orchestra.run.smithi165.stdout:2325 trunc from 0xe2a656f to 0x9121e17 2024-09-11T05:51:34.009 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:34.005+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:34.026 INFO:teuthology.orchestra.run.smithi165.stdout:2326 trunc from 0x9121e17 to 0x4d5bba3 2024-09-11T05:51:34.054 INFO:teuthology.orchestra.run.smithi165.stdout:2327 write 0x8a38fb3 thru 0x8a3f85f (0x68ad bytes) 2024-09-11T05:51:34.057 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.071 INFO:teuthology.orchestra.run.smithi165.stdout:2329 write 0xd9390c0 thru 0xd93cc64 (0x3ba5 bytes) 2024-09-11T05:51:34.075 INFO:teuthology.orchestra.run.smithi165.stdout:2330 punch from 0xaccd214 to 0xacd0c33, (0x3a1f bytes) 2024-09-11T05:51:34.075 INFO:teuthology.orchestra.run.smithi165.stdout:2331 write 0x2a8a4be thru 0x2a95299 (0xaddc bytes) 2024-09-11T05:51:34.095 INFO:teuthology.orchestra.run.smithi165.stdout:2332 trunc from 0xd93cc65 to 0x1b78a1a 2024-09-11T05:51:34.109 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:34.105+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:34.128 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.128 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.136 INFO:teuthology.orchestra.run.smithi165.stdout:2333 trunc from 0x1b78a1a to 0x9d38ea2 2024-09-11T05:51:34.146 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:34.158 INFO:teuthology.orchestra.run.smithi165.stdout:2335 writesame 0xbf4e75e thru 0xbf53b93 data_size 0x3(0x5436 bytes) 2024-09-11T05:51:34.161 INFO:teuthology.orchestra.run.smithi165.stdout:2337 read 0xb6b8a3b thru 0xb6bcc0b (0x41d1 bytes) 2024-09-11T05:51:34.173 INFO:teuthology.orchestra.run.smithi165.stdout:2339 trunc from 0xbf53b94 to 0x4ff7e69 2024-09-11T05:51:34.196 INFO:teuthology.orchestra.run.smithi165.stdout:2342 trunc from 0x4ff7e69 to 0xbd804d 2024-09-11T05:51:34.213 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:34.209+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:34.213 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.240 INFO:teuthology.orchestra.run.smithi165.stdout:2345 write 0xa9be0e8 thru 0xa9be301 (0x21a bytes) 2024-09-11T05:51:34.242 INFO:teuthology.orchestra.run.smithi165.stdout:2346 read 0x2588127 thru 0x259388c (0xb766 bytes) 2024-09-11T05:51:34.243 INFO:teuthology.orchestra.run.smithi165.stdout:2347 punch from 0x6eb1a91 to 0x6ebc17d, (0xa6ec bytes) 2024-09-11T05:51:34.261 INFO:teuthology.orchestra.run.smithi165.stdout:2348 trunc from 0xa9be302 to 0x38e12c 2024-09-11T05:51:34.283 INFO:teuthology.orchestra.run.smithi165.stdout:2349 trunc from 0x38e12c to 0x35434a6 2024-09-11T05:51:34.293 INFO:teuthology.orchestra.run.smithi165.stdout:2350 punch from 0x834fdd to 0x838efb, (0x3f1e bytes) 2024-09-11T05:51:34.293 INFO:teuthology.orchestra.run.smithi165.stdout:2351 punch from 0x498979 to 0x4a5efc, (0xd583 bytes) 2024-09-11T05:51:34.293 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.298 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.298 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.304 INFO:teuthology.orchestra.run.smithi165.stdout:2354 write 0x41959c6 thru 0x41a3fe8 (0xe623 bytes) 2024-09-11T05:51:34.313 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:34.309+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:34.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:34.334 INFO:teuthology.orchestra.run.smithi165.stdout:2355 write 0xe2ac9a5 thru 0xe2b4055 (0x76b1 bytes) 2024-09-11T05:51:34.342 INFO:teuthology.orchestra.run.smithi165.stdout:2356 write 0xc121673 thru 0xc1253bc (0x3d4a bytes) 2024-09-11T05:51:34.348 INFO:teuthology.orchestra.run.smithi165.stdout:2358 punch from 0x2cffba0 to 0x2d09e34, (0xa294 bytes) 2024-09-11T05:51:34.348 INFO:teuthology.orchestra.run.smithi165.stdout:2360 write 0xab6ed88 thru 0xab78772 (0x99eb bytes) 2024-09-11T05:51:34.351 INFO:teuthology.orchestra.run.smithi165.stdout:2361 write 0xb131358 thru 0xb13fab2 (0xe75b bytes) 2024-09-11T05:51:34.374 INFO:teuthology.orchestra.run.smithi165.stdout:2362 trunc from 0xe2b4056 to 0xd580ac 2024-09-11T05:51:34.398 INFO:teuthology.orchestra.run.smithi165.stdout:2364 punch from 0x10685f to 0x111426, (0xabc7 bytes) 2024-09-11T05:51:34.398 INFO:teuthology.orchestra.run.smithi165.stdout:2365 punch from 0x111a0f to 0x11346c, (0x1a5d bytes) 2024-09-11T05:51:34.398 INFO:teuthology.orchestra.run.smithi165.stdout:2366 read 0xdda3f thru 0xdef4e (0x1510 bytes) 2024-09-11T05:51:34.399 INFO:teuthology.orchestra.run.smithi165.stdout:2368 read 0x4ff429 thru 0x506689 (0x7261 bytes) 2024-09-11T05:51:34.400 INFO:teuthology.orchestra.run.smithi165.stdout:2369 write 0x836feb thru 0x840416 (0x942c bytes) 2024-09-11T05:51:34.414 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:34.413+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:34.415 INFO:teuthology.orchestra.run.smithi165.stdout:2370 writesame 0x3455932 thru 0x3455c64 data_size 0x1(0x333 bytes) 2024-09-11T05:51:34.449 INFO:teuthology.orchestra.run.smithi165.stdout:2371 writesame 0xea46854 thru 0xea4e3e5 data_size 0x1(0x7b92 bytes) 2024-09-11T05:51:34.449 INFO:teuthology.orchestra.run.smithi165.stdout:2372 write 0x998f18e thru 0x9995d2d (0x6ba0 bytes) 2024-09-11T05:51:34.453 INFO:teuthology.orchestra.run.smithi165.stdout:2373 read 0x4d13aaf thru 0x4d2023a (0xc78c bytes) 2024-09-11T05:51:34.453 INFO:teuthology.orchestra.run.smithi165.stdout:2374 write 0x93f4921 thru 0x94033ed (0xeacd bytes) 2024-09-11T05:51:34.466 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.466 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.476 INFO:teuthology.orchestra.run.smithi165.stdout:2376 trunc from 0xea4e3e6 to 0x45b315b 2024-09-11T05:51:34.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:34.501 INFO:teuthology.orchestra.run.smithi165.stdout:2377 write 0x4dcfbdb thru 0x4dd9b54 (0x9f7a bytes) 2024-09-11T05:51:34.504 INFO:teuthology.orchestra.run.smithi165.stdout:2378 punch from 0x424929f to 0x4250b0c, (0x786d bytes) 2024-09-11T05:51:34.504 INFO:teuthology.orchestra.run.smithi165.stdout:2380 writesame 0x2a75694 thru 0x2a7c5b3 data_size 0x70(0x6f20 bytes) 2024-09-11T05:51:34.514 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:34.509+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:34.520 INFO:teuthology.orchestra.run.smithi165.stdout:2382 write 0x8b3281b thru 0x8b36d21 (0x4507 bytes) 2024-09-11T05:51:34.523 INFO:teuthology.orchestra.run.smithi165.stdout:2383 read 0x7c96a9b thru 0x7c9aea0 (0x4406 bytes) 2024-09-11T05:51:34.524 INFO:teuthology.orchestra.run.smithi165.stdout:2384 read 0x182aaf9 thru 0x18371e8 (0xc6f0 bytes) 2024-09-11T05:51:34.536 INFO:teuthology.orchestra.run.smithi165.stdout:2385 write 0xba42d35 thru 0xba51f14 (0xf1e0 bytes) 2024-09-11T05:51:34.553 INFO:teuthology.orchestra.run.smithi165.stdout:2386 write 0xe0efc55 thru 0xe0f9e65 (0xa211 bytes) 2024-09-11T05:51:34.558 INFO:teuthology.orchestra.run.smithi165.stdout:2387 read 0x3816eaf thru 0x382602d (0xf17f bytes) 2024-09-11T05:51:34.559 INFO:teuthology.orchestra.run.smithi165.stdout:2389 read 0xa49db94 thru 0xa49ee84 (0x12f1 bytes) 2024-09-11T05:51:34.560 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.560 INFO:teuthology.orchestra.run.smithi165.stdout:2391 read 0xc1ba105 thru 0xc1c3566 (0x9462 bytes) 2024-09-11T05:51:34.560 INFO:teuthology.orchestra.run.smithi165.stdout:2392 writesame 0x35e6f63 thru 0x35f381c data_size 0x2(0xc8ba bytes) 2024-09-11T05:51:34.572 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.572 INFO:teuthology.orchestra.run.smithi165.stdout:2394 write 0x872f9a8 thru 0x8735a09 (0x6062 bytes) 2024-09-11T05:51:34.575 INFO:teuthology.orchestra.run.smithi165.stdout:2395 punch from 0xdd64319 to 0xdd6c7f0, (0x84d7 bytes) 2024-09-11T05:51:34.586 INFO:teuthology.orchestra.run.smithi165.stdout:2397 trunc from 0xe0f9e66 to 0x6e26f88 2024-09-11T05:51:34.614 INFO:teuthology.orchestra.run.smithi165.stdout:2398 trunc from 0x6e26f88 to 0x93f486d 2024-09-11T05:51:34.615 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:34.613+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:34.623 INFO:teuthology.orchestra.run.smithi165.stdout:2399 read 0x779d7eb thru 0x779ed33 (0x1549 bytes) 2024-09-11T05:51:34.624 INFO:teuthology.orchestra.run.smithi165.stdout:2401 write 0x386dd88 thru 0x3872e44 (0x50bd bytes) 2024-09-11T05:51:34.628 INFO:teuthology.orchestra.run.smithi165.stdout:2402 punch from 0x32d520f to 0x32e0258, (0xb049 bytes) 2024-09-11T05:51:34.628 INFO:teuthology.orchestra.run.smithi165.stdout:2403 read 0x4e67e95 thru 0x4e7537e (0xd4ea bytes) 2024-09-11T05:51:34.629 INFO:teuthology.orchestra.run.smithi165.stdout:2405 write 0x64cb3bd thru 0x64cec17 (0x385b bytes) 2024-09-11T05:51:34.638 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.638 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.646 INFO:teuthology.orchestra.run.smithi165.stdout:2406 write 0xbb3ec84 thru 0xbb4e654 (0xf9d1 bytes) 2024-09-11T05:51:34.650 INFO:teuthology.orchestra.run.smithi165.stdout:2408 writesame 0xae8cd22 thru 0xae8d89f data_size 0x1(0xb7e bytes) 2024-09-11T05:51:34.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:34.656 INFO:teuthology.orchestra.run.smithi165.stdout:2409 write 0x6127c8a thru 0x612ff1e (0x8295 bytes) 2024-09-11T05:51:34.658 INFO:teuthology.orchestra.run.smithi165.stdout:2410 punch from 0x4db0be1 to 0x4dbc1c2, (0xb5e1 bytes) 2024-09-11T05:51:34.658 INFO:teuthology.orchestra.run.smithi165.stdout:2411 read 0x986bba2 thru 0x986e48c (0x28eb bytes) 2024-09-11T05:51:34.659 INFO:teuthology.orchestra.run.smithi165.stdout:2412 writesame 0x295f9a0 thru 0x29600e6 data_size 0x9(0x747 bytes) 2024-09-11T05:51:34.663 INFO:teuthology.orchestra.run.smithi165.stdout:2414 punch from 0x1475c4 to 0x150e7a, (0x98b6 bytes) 2024-09-11T05:51:34.663 INFO:teuthology.orchestra.run.smithi165.stdout:2415 write 0x4340489 thru 0x434ee19 (0xe991 bytes) 2024-09-11T05:51:34.680 INFO:teuthology.orchestra.run.smithi165.stdout:2417 writesame 0xd156c95 thru 0xd1660e3 data_size 0x11(0xf44f bytes) 2024-09-11T05:51:34.680 INFO:teuthology.orchestra.run.smithi165.stdout:2419 read 0x1fa0d40 thru 0x1fb0471 (0xf732 bytes) 2024-09-11T05:51:34.681 INFO:teuthology.orchestra.run.smithi165.stdout:2420 punch from 0xef7f19 to 0xeff97c, (0x7a63 bytes) 2024-09-11T05:51:34.708 INFO:teuthology.orchestra.run.smithi165.stdout:2421 trunc from 0xd1660e4 to 0x70e3ba 2024-09-11T05:51:34.715 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:34.713+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:34.724 INFO:teuthology.orchestra.run.smithi165.stdout:2424 read 0x1a78fc thru 0x1a8e0b (0x1510 bytes) 2024-09-11T05:51:34.726 INFO:teuthology.orchestra.run.smithi165.stdout:2425 read 0x1ef9b2 thru 0x1f07ea (0xe39 bytes) 2024-09-11T05:51:34.736 INFO:teuthology.orchestra.run.smithi165.stdout:2426 writesame 0xbfd0d9 thru 0xc045d4 data_size 0x2(0x74fc bytes) 2024-09-11T05:51:34.738 INFO:teuthology.orchestra.run.smithi165.stdout:2427 read 0x7f76ba thru 0x7f9a16 (0x235d bytes) 2024-09-11T05:51:34.751 INFO:teuthology.orchestra.run.smithi165.stdout:2428 writesame 0x34838fa thru 0x348b689 data_size 0x38(0x7d90 bytes) 2024-09-11T05:51:34.753 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.753 INFO:teuthology.orchestra.run.smithi165.stdout:2431 punch from 0x267467e to 0x2683630, (0xefb2 bytes) 2024-09-11T05:51:34.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:51:34.774 INFO:teuthology.orchestra.run.smithi165.stdout:2432 writesame 0xa0e6eca thru 0xa0f398a data_size 0x7(0xcac1 bytes) 2024-09-11T05:51:34.788 INFO:teuthology.orchestra.run.smithi165.stdout:2433 write 0xccaf81a thru 0xccbaf38 (0xb71f bytes) 2024-09-11T05:51:34.790 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.790 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.802 INFO:teuthology.orchestra.run.smithi165.stdout:2434 writesame 0xd643efc thru 0xd6528c4 data_size 0x1(0xe9c9 bytes) 2024-09-11T05:51:34.802 INFO:teuthology.orchestra.run.smithi165.stdout:2435 punch from 0xca578b0 to 0xca5c170, (0x48c0 bytes) 2024-09-11T05:51:34.803 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.803 INFO:teuthology.orchestra.run.smithi165.stdout:2439 writesame 0xc413672 thru 0xc4174af data_size 0x1f(0x3e3e bytes) 2024-09-11T05:51:34.808 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.808 INFO:teuthology.orchestra.run.smithi165.stdout:2442 read 0xbfe5d5f thru 0xbfe69e7 (0xc89 bytes) 2024-09-11T05:51:34.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:34.809 INFO:teuthology.orchestra.run.smithi165.stdout:2443 writesame 0x1eee7cb thru 0x1efae2c data_size 0x43(0xc662 bytes) 2024-09-11T05:51:34.813 INFO:teuthology.orchestra.run.smithi165.stdout:2444 read 0x97aed5b thru 0x97bec39 (0xfedf bytes) 2024-09-11T05:51:34.814 INFO:teuthology.orchestra.run.smithi165.stdout:2445 writesame 0xa98a093 thru 0xa9968c5 data_size 0x1(0xc833 bytes) 2024-09-11T05:51:34.815 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:34.813+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:34.851 INFO:teuthology.orchestra.run.smithi165.stdout:2446 read 0x5e620e6 thru 0x5e6b894 (0x97af bytes) 2024-09-11T05:51:34.852 INFO:teuthology.orchestra.run.smithi165.stdout:2447 read 0x980b391 thru 0x9813deb (0x8a5b bytes) 2024-09-11T05:51:34.853 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.853 INFO:teuthology.orchestra.run.smithi165.stdout:2449 writesame 0x55126f5 thru 0x551a006 data_size 0x1(0x7912 bytes) 2024-09-11T05:51:34.896 INFO:teuthology.orchestra.run.smithi165.stdout:2450 write 0x3f02ead thru 0x3f0d65f (0xa7b3 bytes) 2024-09-11T05:51:34.897 INFO:teuthology.orchestra.run.smithi165.stdout:2452 write 0x88dc405 thru 0x88e3197 (0x6d93 bytes) 2024-09-11T05:51:34.903 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.903 INFO:teuthology.orchestra.run.smithi165.stdout:2454 writesame 0x15d225d thru 0x15dbb9f data_size 0x23(0x9943 bytes) 2024-09-11T05:51:34.907 INFO:teuthology.orchestra.run.smithi165.stdout:2455 read 0x4621a83 thru 0x462d44b (0xb9c9 bytes) 2024-09-11T05:51:34.916 INFO:teuthology.orchestra.run.smithi165.stdout:2456 trunc from 0xd6528c5 to 0x7dad292 2024-09-11T05:51:34.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:34.917 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:34.917+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:34.936 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:34.937 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:34.938 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:34.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:34.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:51:34.951 INFO:teuthology.orchestra.run.smithi165.stdout:2457 trunc from 0x7dad292 to 0xcffaa0e 2024-09-11T05:51:34.959 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:34.959 INFO:teuthology.orchestra.run.smithi165.stdout:2461 punch from 0xb1aa549 to 0xb1b3ba7, (0x965e bytes) 2024-09-11T05:51:34.960 INFO:teuthology.orchestra.run.smithi165.stdout:2462 writesame 0x29660e6 thru 0x2972028 data_size 0x13(0xbf43 bytes) 2024-09-11T05:51:34.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:34.964 INFO:teuthology.orchestra.run.smithi165.stdout:2463 punch from 0xaa53547 to 0xaa5e279, (0xad32 bytes) 2024-09-11T05:51:34.964 INFO:teuthology.orchestra.run.smithi165.stdout:2464 read 0x6b9633c thru 0x6ba186d (0xb532 bytes) 2024-09-11T05:51:34.964 INFO:teuthology.orchestra.run.smithi165.stdout:2465 read 0xd90173 thru 0xd9ab28 (0xa9b6 bytes) 2024-09-11T05:51:34.965 INFO:teuthology.orchestra.run.smithi165.stdout:2466 read 0xb5f1fd9 thru 0xb5fca3f (0xaa67 bytes) 2024-09-11T05:51:34.966 INFO:teuthology.orchestra.run.smithi165.stdout:2467 punch from 0x6124098 to 0x6131e66, (0xddce bytes) 2024-09-11T05:51:34.966 INFO:teuthology.orchestra.run.smithi165.stdout:2468 punch from 0x29b630c to 0x29c1b9e, (0xb892 bytes) 2024-09-11T05:51:34.966 INFO:teuthology.orchestra.run.smithi165.stdout:2469 read 0xce7a0c3 thru 0xce8258f (0x84cd bytes) 2024-09-11T05:51:34.967 INFO:teuthology.orchestra.run.smithi165.stdout:2470 trunc from 0xcffaa0e to 0xc902df9 2024-09-11T05:51:34.976 INFO:teuthology.orchestra.run.smithi165.stdout:2471 read 0xa3a9675 thru 0xa3b8fb4 (0xf940 bytes) 2024-09-11T05:51:34.978 INFO:teuthology.orchestra.run.smithi165.stdout:2472 trunc from 0xc902df9 to 0xd0fbfeb 2024-09-11T05:51:34.986 INFO:teuthology.orchestra.run.smithi165.stdout:2473 write 0x8a98994 thru 0x8a993d5 (0xa42 bytes) 2024-09-11T05:51:34.988 INFO:teuthology.orchestra.run.smithi165.stdout:2474 read 0x7840c75 thru 0x7845901 (0x4c8d bytes) 2024-09-11T05:51:34.989 INFO:teuthology.orchestra.run.smithi165.stdout:2475 write 0x852fdee thru 0x853b92b (0xbb3e bytes) 2024-09-11T05:51:34.993 INFO:teuthology.orchestra.run.smithi165.stdout:2476 read 0x4f4815c thru 0x4f4da22 (0x58c7 bytes) 2024-09-11T05:51:34.994 INFO:teuthology.orchestra.run.smithi165.stdout:2477 read 0xaab3eb0 thru 0xaabcb09 (0x8c5a bytes) 2024-09-11T05:51:34.995 INFO:teuthology.orchestra.run.smithi165.stdout:2478 read 0xc7c9bbd thru 0xc7d18bd (0x7d01 bytes) 2024-09-11T05:51:34.996 INFO:teuthology.orchestra.run.smithi165.stdout:2479 write 0x5dc0c0c thru 0x5dc2d3a (0x212f bytes) 2024-09-11T05:51:34.997 INFO:teuthology.orchestra.run.smithi165.stdout:2481 punch from 0xa272e5d to 0xa2805d2, (0xd775 bytes) 2024-09-11T05:51:34.997 INFO:teuthology.orchestra.run.smithi165.stdout:2483 punch from 0x5176e42 to 0x517ed23, (0x7ee1 bytes) 2024-09-11T05:51:35.007 INFO:teuthology.orchestra.run.smithi165.stdout:2485 write 0xde85b22 thru 0xde94fdc (0xf4bb bytes) 2024-09-11T05:51:35.015 INFO:teuthology.orchestra.run.smithi165.stdout:2486 punch from 0xd6ba685 to 0xd6c1457, (0x6dd2 bytes) 2024-09-11T05:51:35.015 INFO:teuthology.orchestra.run.smithi165.stdout:2487 writesame 0xca42756 thru 0xca4ca09 data_size 0x9c(0xa2b4 bytes) 2024-09-11T05:51:35.018 INFO:teuthology.orchestra.run.smithi165.stdout:2488 write 0xc039c46 thru 0xc047794 (0xdb4f bytes) 2024-09-11T05:51:35.018 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:35.017+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:35.026 INFO:teuthology.orchestra.run.smithi165.stdout:2489 read 0xca0e277 thru 0xca0f4c8 (0x1252 bytes) 2024-09-11T05:51:35.035 INFO:teuthology.orchestra.run.smithi165.stdout:2490 write 0xe3cd3c3 thru 0xe3da99c (0xd5da bytes) 2024-09-11T05:51:35.049 INFO:teuthology.orchestra.run.smithi165.stdout:2491 trunc from 0xe3da99d to 0x876219c 2024-09-11T05:51:35.074 INFO:teuthology.orchestra.run.smithi165.stdout:2492 punch from 0x990ef5 to 0x998cc0, (0x7dcb bytes) 2024-09-11T05:51:35.090 INFO:teuthology.orchestra.run.smithi165.stdout:2493 write 0xcef017b thru 0xcef8d6e (0x8bf4 bytes) 2024-09-11T05:51:35.097 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:35.097 INFO:teuthology.orchestra.run.smithi165.stdout:2496 punch from 0x7f93134 to 0x7f982fb, (0x51c7 bytes) 2024-09-11T05:51:35.097 INFO:teuthology.orchestra.run.smithi165.stdout:2497 read 0x5933f53 thru 0x593e497 (0xa545 bytes) 2024-09-11T05:51:35.108 INFO:teuthology.orchestra.run.smithi165.stdout:2499 writesame 0xd7618b3 thru 0xd770f33 data_size 0x23(0xf681 bytes) 2024-09-11T05:51:35.108 INFO:teuthology.orchestra.run.smithi165.stdout:2500 read 0xba9a6ea thru 0xbaaa142 (0xfa59 bytes) 2024-09-11T05:51:35.108 INFO:teuthology.orchestra.run.smithi165.stdout:2501 read 0x4a96f04 thru 0x4aa3341 (0xc43e bytes) 2024-09-11T05:51:35.110 INFO:teuthology.orchestra.run.smithi165.stdout:2502 write 0x920d51f thru 0x9212243 (0x4d25 bytes) 2024-09-11T05:51:35.113 INFO:teuthology.orchestra.run.smithi165.stdout:2503 punch from 0x9179051 to 0x9185e16, (0xcdc5 bytes) 2024-09-11T05:51:35.113 INFO:teuthology.orchestra.run.smithi165.stdout:2506 punch from 0x99b4a51 to 0x99bf6dc, (0xac8b bytes) 2024-09-11T05:51:35.113 INFO:teuthology.orchestra.run.smithi165.stdout:2507 write 0xca02fa4 thru 0xca06d33 (0x3d90 bytes) 2024-09-11T05:51:35.115 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:35.115 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:35.119 INFO:teuthology.orchestra.run.smithi165.stdout:2508 write 0x4d2ca53 thru 0x4d2fbca (0x3178 bytes) 2024-09-11T05:51:35.119 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:35.117+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:35.121 INFO:teuthology.orchestra.run.smithi165.stdout:2509 writesame 0x8215aae thru 0x821ba3d data_size 0x16(0x5f90 bytes) 2024-09-11T05:51:35.126 INFO:teuthology.orchestra.run.smithi165.stdout:2511 write 0x93ef91a thru 0x93f1d1f (0x2406 bytes) 2024-09-11T05:51:35.127 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:35.127 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:35.130 INFO:teuthology.orchestra.run.smithi165.stdout:2512 punch from 0x61150a9 to 0x612037c, (0xb2d3 bytes) 2024-09-11T05:51:35.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:35.147 INFO:teuthology.orchestra.run.smithi165.stdout:2513 trunc from 0xd770f34 to 0x180c01d 2024-09-11T05:51:35.150 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:35.150 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:35.171 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:35.171 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:35.172 INFO:teuthology.orchestra.run.smithi165.stdout:2514 read 0x105d18f thru 0x1069f39 (0xcdab bytes) 2024-09-11T05:51:35.189 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:35.189 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:35.195 INFO:teuthology.orchestra.run.smithi165.stdout:2516 write 0xa6b8736 thru 0xa6c81f2 (0xfabd bytes) 2024-09-11T05:51:35.199 INFO:teuthology.orchestra.run.smithi165.stdout:2518 writesame 0x5980a3b thru 0x5982c16 data_size 0x16(0x21dc bytes) 2024-09-11T05:51:35.203 INFO:teuthology.orchestra.run.smithi165.stdout:2519 writesame 0x14a97df thru 0x14aed70 data_size 0x9(0x5592 bytes) 2024-09-11T05:51:35.210 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:35.210 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:35.220 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:35.221+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:35.223 INFO:teuthology.orchestra.run.smithi165.stdout:2520 trunc from 0xa6c81f3 to 0xea0cbb 2024-09-11T05:51:35.233 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:35.234 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:35.239 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:35.255 INFO:teuthology.orchestra.run.smithi165.stdout:2522 writesame 0x5e12860 thru 0x5e12d3f data_size 0x10(0x4e0 bytes) 2024-09-11T05:51:35.256 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:35.256 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:35.264 INFO:teuthology.orchestra.run.smithi165.stdout:2523 trunc from 0x5e12d40 to 0x975f669 2024-09-11T05:51:35.273 INFO:teuthology.orchestra.run.smithi165.stdout:2525 punch from 0x388d3f1 to 0x3896462, (0x9071 bytes) 2024-09-11T05:51:35.273 INFO:teuthology.orchestra.run.smithi165.stdout:2526 write 0x7685f48 thru 0x7691f77 (0xc030 bytes) 2024-09-11T05:51:35.276 INFO:teuthology.orchestra.run.smithi165.stdout:2527 read 0x184bbb1 thru 0x1850119 (0x4569 bytes) 2024-09-11T05:51:35.279 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:35.280 INFO:teuthology.orchestra.run.smithi165.stdout:2529 writesame 0x5cfc5e2 thru 0x5d04e81 data_size 0x4(0x88a0 bytes) 2024-09-11T05:51:35.280 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:35.287 INFO:teuthology.orchestra.run.smithi165.stdout:2531 read 0x470eea2 thru 0x471e5bd (0xf71c bytes) 2024-09-11T05:51:35.288 INFO:teuthology.orchestra.run.smithi165.stdout:2533 read 0x2c2d1e0 thru 0x2c36927 (0x9748 bytes) 2024-09-11T05:51:35.293 INFO:teuthology.orchestra.run.smithi165.stdout:2535 trunc from 0x975f669 to 0x6845e44 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.081511885000001, 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:35.295 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:35.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:35.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.299 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978522154, 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.883980118, 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.304 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:35.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:35.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:35.308 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.612232834, 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:35.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.488213859999998, 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:35.313 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:35.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.317 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6354 3.9 e33/23 3:9445ea4e:::rbd_data.11249b359231.000000000000003b:head v 33'25, mlcod=33'23)", 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.476840687999999, 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026120029999999999, 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816608+0000", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816609+0000", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2139999999999999e-06 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816617+0000", 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7680000000000008e-06 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816630+0000", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816637+0000", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9260000000000003e-06 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816690+0000", 2024-09-11T05:51:35.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2533e-05 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.816706+0000", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6407000000000001e-05 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819170+0000", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024645309999999998 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.819220+0000", 2024-09-11T05:51:35.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9595999999999997e-05 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.413221684, 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:35.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.939246564000001, 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:35.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:35.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.4466173, 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:35.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:35.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7116 3.9 e34/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 34'35, mlcod=34'33)", 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.286389323000002, 2024-09-11T05:51:35.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00371405, 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007061+0000", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5090000000000001e-06 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007072+0000", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1399e-05 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007100+0000", 2024-09-11T05:51:35.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7531000000000001e-05 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007114+0000", 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4378e-05 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007340+0000", 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022568499999999999 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007387+0000", 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7194000000000002e-05 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010717+0000", 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033296369999999999 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010773+0000", 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6716999999999999e-05 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7119 3.4 e34/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 34'49, mlcod=34'47)", 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.285183544999999, 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034162110000000002, 2024-09-11T05:51:35.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008267+0000", 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.294e-06 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008276+0000", 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5820000000000008e-06 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008290+0000", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4005000000000001e-05 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008298+0000", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3189999999999995e-06 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008368+0000", 2024-09-11T05:51:35.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0055999999999995e-05 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008396+0000", 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7563999999999999e-05 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011628+0000", 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032314969999999998 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011681+0000", 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3894000000000003e-05 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12734 3.0 e38/23 3:06e59eda:::rbd_data.1124518b2c12.000000000000002b:head v 38'41, mlcod=37'39)", 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.483173541999999, 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033376479999999999, 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.345 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810277+0000", 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810314+0000", 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7370999999999998e-05 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810336+0000", 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2129000000000001e-05 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810347+0000", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0465999999999999e-05 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810418+0000", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099000000000005e-05 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810443+0000", 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4825999999999999e-05 2024-09-11T05:51:35.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813553+0000", 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031099790000000001 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813613+0000", 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0307999999999999e-05 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15206 3.0 e41/23 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:head v 41'65, mlcod=41'63)", 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.2894733110000001, 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034631129999999999, 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003977+0000", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003986+0000", 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004008+0000", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1679e-05 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004018+0000", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5210000000000008e-06 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004105+0000", 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7371000000000001e-05 2024-09-11T05:51:35.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004128+0000", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2682999999999999e-05 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007395+0000", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032676129999999999 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007439+0000", 2024-09-11T05:51:35.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3306999999999997e-05 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15537 3.4 e42/23 3:242e3831:::rbd_data.11243bbd48bb.000000000000001e:head v 42'64, mlcod=41'61)", 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.5303539619999995, 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031185620000000001, 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763096+0000", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763105+0000", 2024-09-11T05:51:35.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2119999999999992e-06 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763125+0000", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9536999999999999e-05 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763135+0000", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0022e-05 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763216+0000", 2024-09-11T05:51:35.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1606000000000004e-05 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763236+0000", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9585e-05 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766184+0000", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002948169 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766213+0000", 2024-09-11T05:51:35.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.917e-05 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15715 3.0 e42/23 3:0151ec39:::rbd_data.1124d6faf06d.000000000000006a:head v 42'71, mlcod=42'67)", 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.373645368, 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028945469999999999, 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919804+0000", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-07 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919810+0000", 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8919999999999997e-06 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919823+0000", 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2879e-05 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919830+0000", 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.635e-06 2024-09-11T05:51:35.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919877+0000", 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7049999999999998e-05 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919896+0000", 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9216999999999999e-05 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922658+0000", 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002762675 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922698+0000", 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9477000000000002e-05 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:16482 3.9 e43/23 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:head v 43'50, mlcod=40'48)", 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.4899086300000004, 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046884810000000004, 2024-09-11T05:51:35.362 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803541+0000", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8899999999999999e-07 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.804985+0000", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014445459999999999 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805024+0000", 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8390999999999999e-05 2024-09-11T05:51:35.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805036+0000", 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799e-05 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805107+0000", 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1530000000000001e-05 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805135+0000", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8384000000000001e-05 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808057+0000", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029216979999999999 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808229+0000", 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017144400000000001 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.366 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(3.9 epoch 47/23 rep_tid 602 entries 47'74 (0'0) error 3:957a1c2c:::rbd_data.11244d01bf21.0000000000000016:head by client.4388.0:19596 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 46'72)", 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.3573810210000001, 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030496590000000001, 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:35.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.31e-07 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936074+0000", 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1870000000000002e-06 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936127+0000", 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3325999999999997e-05 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936151+0000", 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3354e-05 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936266+0000", 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115061 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.939117+0000", 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028516000000000001 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.370 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:19761 3.9 e47/23 3:97eabedf:::rbd_data.11244d01bf21.0000000000000021:head v 47'76, mlcod=46'73)", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.349425214, 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051400339999999999, 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944025+0000", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944036+0000", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0757999999999999e-05 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944052+0000", 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6565000000000001e-05 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944061+0000", 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7479999999999999e-06 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944130+0000", 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9088e-05 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944154+0000", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4541000000000001e-05 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.949044+0000", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048898650000000002 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.949164+0000", 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119232 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:19811 3.0 e47/23 3:0029ad0d:::rbd_data.11244d01bf21.0000000000000033:head v 47'174, mlcod=46'171)", 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.962174+0000", 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.3312748499999998, 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002805381, 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962174+0000", 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962174+0000", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962174+0000", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.91e-07 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962180+0000", 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2140000000000001e-06 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962199+0000", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8374999999999999e-05 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962208+0000", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7739999999999993e-06 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962301+0000", 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3305000000000002e-05 2024-09-11T05:51:35.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.962316+0000", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5562000000000001e-05 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.964932+0000", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002615254 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.964979+0000", 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7605999999999998e-05 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:19824 3.9 e47/23 3:971d1760:::rbd_data.11244d01bf21.000000000000003c:head v 47'81, mlcod=47'77)", 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.3221788970000001, 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035938300000000001, 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971271+0000", 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.102e-06 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971279+0000", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8790000000000003e-06 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971298+0000", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8771e-05 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971307+0000", 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-11T05:51:35.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971394+0000", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6780000000000006e-05 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971424+0000", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.082e-05 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.974820+0000", 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003395726 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.974864+0000", 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3550999999999997e-05 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.383 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:35.384 INFO:teuthology.orchestra.run.smithi165.stdout:2537 read 0x5616b21 thru 0x56269f7 (0xfed7 bytes) 2024-09-11T05:51:35.384 INFO:teuthology.orchestra.run.smithi165.stdout:2538 write 0x64930e2 thru 0x64a1871 (0xe790 bytes) 2024-09-11T05:51:35.385 INFO:teuthology.orchestra.run.smithi165.stdout:2539 read 0x10b8b0a thru 0x10c1353 (0x884a bytes) 2024-09-11T05:51:35.385 INFO:teuthology.orchestra.run.smithi165.stdout:2541 read 0x3f57868 thru 0x3f5ff21 (0x86ba bytes) 2024-09-11T05:51:35.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:35.387 INFO:teuthology.orchestra.run.smithi165.stdout:2542 writesame 0xe0bf6fd thru 0xe0c0592 data_size 0x1(0xe96 bytes) 2024-09-11T05:51:35.387 INFO:teuthology.orchestra.run.smithi165.stdout:2543 write 0xddf9dc3 thru 0xddfc5dd (0x281b bytes) 2024-09-11T05:51:35.387 INFO:teuthology.orchestra.run.smithi165.stdout:2544 write 0x922481e thru 0x922e047 (0x982a bytes) 2024-09-11T05:51:35.387 INFO:teuthology.orchestra.run.smithi165.stdout:2545 clone 14 order 20 su 32768 sc 4 2024-09-11T05:51:35.388 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:35.389 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:51:35.389 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-11T05:51:34.502587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":299849873,"num_objects":523,"num_object_clones":201,"num_object_copies":1046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":177,"num_read":32392,"num_read_kb":111634,"num_write":5589,"num_write_kb":70996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4429,"ondisk_log_size":4429,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":313880,"kb_used_data":60212,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754660840,"statfs":{"total":773094113280,"available":772772700160,"internally_reserved":0,"allocated":61657088,"data_stored":92902920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[30,13],"upper_bound":4},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53004388,"num_objects":247,"num_object_clones":93,"num_object_copies":494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":104,"num_read":9167,"num_read_kb":29087,"num_write":1782,"num_write_kb":24817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.816113"},"pg_stats":[{"pgid":"3.1f","version":"44'177","reported_seq":766,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.690861+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:51:29.690861+0000","last_peered":"2024-09-11T05:51:29.690861+0000","last_clean":"2024-09-11T05:51:29.690861+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:51:29.690861+0000","last_undegraded":"2024-09-11T05:51:29.690861+0000","last_fullsized":"2024-09-11T05:51:29.690861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:50.998246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":4475040,"num_objects":26,"num_object_clones":11,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":9,"num_read":1006,"num_read_kb":1704,"num_write":228,"num_write_kb":917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"48'100","reported_seq":964,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:34.122933+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:51:34.122933+0000","last_peered":"2024-09-11T05:51:34.122933+0000","last_clean":"2024-09-11T05:51:34.122933+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:51:34.122933+0000","last_undegraded":"2024-09-11T05:51:34.122933+0000","last_fullsized":"2024-09-11T05:51:34.122933+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:48:29.938248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":17931350,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":1183,"num_read_kb":3130,"num_write":88,"num_write_kb":600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"44'43","reported_seq":165,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.699662+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:51:29.699662+0000","last_peered":"2024-09-11T05:51:29.699662+0000","last_clean":"2024-09-11T05:51:29.699662+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:51:29.699662+0000","last_undegraded":"2024-09-11T05:51:29.699662+0000","last_fullsized":"2024-09-11T05:51:29.699662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:55:43.741838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":8603994,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":60,"num_read_kb":3775,"num_write":88,"num_write_kb":2126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"45'211","reported_seq":395,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.699553+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:51:29.699553+0000","last_peered":"2024-09-11T05:51:29.699553+0000","last_clean":"2024-09-11T05:51:29.699553+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:51:29.699553+0000","last_undegraded":"2024-09-11T05:51:29.699553+0000","last_fullsized":"2024-09-11T05:51:29.699553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:25:20.314851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":1723495,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":4,"num_read":993,"num_read_kb":851,"num_write":337,"num_write_kb":613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'107","reported_seq":836,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.690875+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:51:29.690875+0000","last_peered":"2024-09-11T05:51:29.690875+0000","last_clean":"2024-09-11T05:51:29.690875+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:51:29.690875+0000","last_undegraded":"2024-09-11T05:51:29.690875+0000","last_fullsized":"2024-09-11T05:51:29.690875+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:42:09.980450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":4886835,"num_objects":21,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":9,"num_read":1105,"num_read_kb":2215,"num_write":94,"num_write_kb":488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"44'108","reported_seq":1390,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.838768+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:51:29.838768+0000","last_peered":"2024-09-11T05:51:29.838768+0000","last_clean":"2024-09-11T05:51:29.838768+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:51:29.838768+0000","last_undegraded":"2024-09-11T05:51:29.838768+0000","last_fullsized":"2024-09-11T05:51:29.838768+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:16.679098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":22595305,"num_objects":21,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":7,"num_read":1374,"num_read_kb":4849,"num_write":143,"num_write_kb":4031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"44'341","reported_seq":770,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.798520+0000","last_change":"2024-09-11T05:51:07.514037+0000","last_active":"2024-09-11T05:51:29.798520+0000","last_peered":"2024-09-11T05:51:29.798520+0000","last_clean":"2024-09-11T05:51:29.798520+0000","last_became_active":"2024-09-11T05:51:05.507475+0000","last_became_peered":"2024-09-11T05:51:05.507475+0000","last_unstale":"2024-09-11T05:51:29.798520+0000","last_undegraded":"2024-09-11T05:51:29.798520+0000","last_fullsized":"2024-09-11T05:51:29.798520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:31:04.162228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":986315,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":2331,"num_read_kb":2491,"num_write":522,"num_write_kb":871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"48'199","reported_seq":1713,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:34.032633+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:51:34.032633+0000","last_peered":"2024-09-11T05:51:34.032633+0000","last_clean":"2024-09-11T05:51:34.032633+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:51:34.032633+0000","last_undegraded":"2024-09-11T05:51:34.032633+0000","last_fullsized":"2024-09-11T05:51:34.032633+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:34.264294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":13087710,"num_objects":28,"num_object_clones":10,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":2782,"num_read_kb":6995,"num_write":226,"num_write_kb":7059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"44'88","reported_seq":383,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.690848+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:51:29.690848+0000","last_peered":"2024-09-11T05:51:29.690848+0000","last_clean":"2024-09-11T05:51:29.690848+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:51:29.690848+0000","last_undegraded":"2024-09-11T05:51:29.690848+0000","last_fullsized":"2024-09-11T05:51:29.690848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:48:44.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":262144,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":4,"num_read":679,"num_read_kb":566,"num_write":99,"num_write_kb":2506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"45'46","reported_seq":416,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.699804+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:51:29.699804+0000","last_peered":"2024-09-11T05:51:29.699804+0000","last_clean":"2024-09-11T05:51:29.699804+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:51:29.699804+0000","last_undegraded":"2024-09-11T05:51:29.699804+0000","last_fullsized":"2024-09-11T05:51:29.699804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:20.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":10447689,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":5,"num_read":318,"num_read_kb":2066,"num_write":40,"num_write_kb":427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"45'53","reported_seq":204,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.717163+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:51:29.717163+0000","last_peered":"2024-09-11T05:51:29.717163+0000","last_clean":"2024-09-11T05:51:29.717163+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:51:29.717163+0000","last_undegraded":"2024-09-11T05:51:29.717163+0000","last_fullsized":"2024-09-11T05:51:29.717163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.609695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":17613934,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":96,"num_read_kb":291,"num_write":51,"num_write_kb":666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"45'226","reported_seq":1693,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.764585+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:51:29.764585+0000","last_peered":"2024-09-11T05:51:29.764585+0000","last_clean":"2024-09-11T05:51:29.764585+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:51:29.764585+0000","last_undegraded":"2024-09-11T05:51:29.764585+0000","last_fullsized":"2024-09-11T05:51:29.764585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:56:26.658758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":13722997,"num_objects":27,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":8,"num_read":2650,"num_read_kb":11354,"num_write":268,"num_write_kb":4769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714466+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:51:29.714466+0000","last_peered":"2024-09-11T05:51:29.714466+0000","last_clean":"2024-09-11T05:51:29.714466+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:51:29.714466+0000","last_undegraded":"2024-09-11T05:51:29.714466+0000","last_fullsized":"2024-09-11T05:51:29.714466+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"45'174","reported_seq":345,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.863986+0000","last_change":"2024-09-11T05:51:07.518766+0000","last_active":"2024-09-11T05:51:29.863986+0000","last_peered":"2024-09-11T05:51:29.863986+0000","last_clean":"2024-09-11T05:51:29.863986+0000","last_became_active":"2024-09-11T05:51:05.499998+0000","last_became_peered":"2024-09-11T05:51:05.499998+0000","last_unstale":"2024-09-11T05:51:29.863986+0000","last_undegraded":"2024-09-11T05:51:29.863986+0000","last_fullsized":"2024-09-11T05:51:29.863986+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:49.235217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":6078543,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":5,"num_read":718,"num_read_kb":2006,"num_write":264,"num_write_kb":1912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.693584+0000","last_change":"2024-09-11T05:50:58.177276+0000","last_active":"2024-09-11T05:51:29.693584+0000","last_peered":"2024-09-11T05:51:29.693584+0000","last_clean":"2024-09-11T05:51:29.693584+0000","last_became_active":"2024-09-11T05:50:56.154414+0000","last_became_peered":"2024-09-11T05:50:56.154414+0000","last_unstale":"2024-09-11T05:51:29.693584+0000","last_undegraded":"2024-09-11T05:51:29.693584+0000","last_fullsized":"2024-09-11T05:51:29.693584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:57:08.417732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'250","reported_seq":784,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:33.267759+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:51:33.267759+0000","last_peered":"2024-09-11T05:51:33.267759+0000","last_clean":"2024-09-11T05:51:33.267759+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:51:33.267759+0000","last_undegraded":"2024-09-11T05:51:33.267759+0000","last_fullsized":"2024-09-11T05:51:33.267759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:01:15.111998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":4128855,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":4,"num_read":1421,"num_read_kb":4226,"num_write":370,"num_write_kb":1054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714505+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:51:29.714505+0000","last_peered":"2024-09-11T05:51:29.714505+0000","last_clean":"2024-09-11T05:51:29.714505+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:51:29.714505+0000","last_undegraded":"2024-09-11T05:51:29.714505+0000","last_fullsized":"2024-09-11T05:51:29.714505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'81","reported_seq":547,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.693386+0000","last_change":"2024-09-11T05:51:07.514746+0000","last_active":"2024-09-11T05:51:29.693386+0000","last_peered":"2024-09-11T05:51:29.693386+0000","last_clean":"2024-09-11T05:51:29.693386+0000","last_became_active":"2024-09-11T05:51:06.470494+0000","last_became_peered":"2024-09-11T05:51:06.470494+0000","last_unstale":"2024-09-11T05:51:29.693386+0000","last_undegraded":"2024-09-11T05:51:29.693386+0000","last_fullsized":"2024-09-11T05:51:29.693386+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:02:29.995831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":8027573,"num_objects":13,"num_object_clones":6,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":429,"num_read_kb":5394,"num_write":79,"num_write_kb":1193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.693417+0000","last_change":"2024-09-11T05:50:58.177342+0000","last_active":"2024-09-11T05:51:29.693417+0000","last_peered":"2024-09-11T05:51:29.693417+0000","last_clean":"2024-09-11T05:51:29.693417+0000","last_became_active":"2024-09-11T05:50:56.479833+0000","last_became_peered":"2024-09-11T05:50:56.479833+0000","last_unstale":"2024-09-11T05:51:29.693417+0000","last_undegraded":"2024-09-11T05:51:29.693417+0000","last_fullsized":"2024-09-11T05:51:29.693417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:49.401902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094461100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"45'152","reported_seq":943,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.797098+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:51:29.797098+0000","last_peered":"2024-09-11T05:51:29.797098+0000","last_clean":"2024-09-11T05:51:29.797098+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:51:29.797098+0000","last_undegraded":"2024-09-11T05:51:29.797098+0000","last_fullsized":"2024-09-11T05:51:29.797098+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:17:30.013257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":7385787,"num_objects":19,"num_object_clones":7,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":1978,"num_read_kb":2390,"num_write":142,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"45'82","reported_seq":825,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.799599+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:51:29.799599+0000","last_peered":"2024-09-11T05:51:29.799599+0000","last_clean":"2024-09-11T05:51:29.799599+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:51:29.799599+0000","last_undegraded":"2024-09-11T05:51:29.799599+0000","last_fullsized":"2024-09-11T05:51:29.799599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:34:33.826262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":4105653,"num_objects":15,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":5,"num_read":717,"num_read_kb":2947,"num_write":78,"num_write_kb":1206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":67,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.699369+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:51:29.699369+0000","last_peered":"2024-09-11T05:51:29.699369+0000","last_clean":"2024-09-11T05:51:29.699369+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:51:29.699369+0000","last_undegraded":"2024-09-11T05:51:29.699369+0000","last_fullsized":"2024-09-11T05:51:29.699369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"45'137","reported_seq":970,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.853574+0000","last_change":"2024-09-11T05:51:07.514375+0000","last_active":"2024-09-11T05:51:29.853574+0000","last_peered":"2024-09-11T05:51:29.853574+0000","last_clean":"2024-09-11T05:51:29.853574+0000","last_became_active":"2024-09-11T05:51:06.471743+0000","last_became_peered":"2024-09-11T05:51:06.471743+0000","last_unstale":"2024-09-11T05:51:29.853574+0000","last_undegraded":"2024-09-11T05:51:29.853574+0000","last_fullsized":"2024-09-11T05:51:29.853574+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:36.623285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":15078688,"num_objects":29,"num_object_clones":11,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":10,"num_read":1161,"num_read_kb":3351,"num_write":131,"num_write_kb":2082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.687876+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:51:29.687876+0000","last_peered":"2024-09-11T05:51:29.687876+0000","last_clean":"2024-09-11T05:51:29.687876+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:51:29.687876+0000","last_undegraded":"2024-09-11T05:51:29.687876+0000","last_fullsized":"2024-09-11T05:51:29.687876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"45'131","reported_seq":619,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.690754+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:51:29.690754+0000","last_peered":"2024-09-11T05:51:29.690754+0000","last_clean":"2024-09-11T05:51:29.690754+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:51:29.690754+0000","last_undegraded":"2024-09-11T05:51:29.690754+0000","last_fullsized":"2024-09-11T05:51:29.690754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:39.180244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":4735221,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":6,"num_read":1251,"num_read_kb":4953,"num_write":165,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714267+0000","last_change":"2024-09-11T05:50:58.467120+0000","last_active":"2024-09-11T05:51:29.714267+0000","last_peered":"2024-09-11T05:51:29.714267+0000","last_clean":"2024-09-11T05:51:29.714267+0000","last_became_active":"2024-09-11T05:50:56.154228+0000","last_became_peered":"2024-09-11T05:50:56.154228+0000","last_unstale":"2024-09-11T05:51:29.714267+0000","last_undegraded":"2024-09-11T05:51:29.714267+0000","last_fullsized":"2024-09-11T05:51:29.714267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:00.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"48'176","reported_seq":921,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:34.032219+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:51:34.032219+0000","last_peered":"2024-09-11T05:51:34.032219+0000","last_clean":"2024-09-11T05:51:34.032219+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:51:34.032219+0000","last_undegraded":"2024-09-11T05:51:34.032219+0000","last_fullsized":"2024-09-11T05:51:34.032219+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:10:46.749522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":8472367,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":1213,"num_read_kb":7209,"num_write":280,"num_write_kb":5724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.699413+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:51:29.699413+0000","last_peered":"2024-09-11T05:51:29.699413+0000","last_clean":"2024-09-11T05:51:29.699413+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:51:29.699413+0000","last_undegraded":"2024-09-11T05:51:29.699413+0000","last_fullsized":"2024-09-11T05:51:29.699413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":215,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.713982+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:51:29.713982+0000","last_peered":"2024-09-11T05:51:29.713982+0000","last_clean":"2024-09-11T05:51:29.713982+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:51:29.713982+0000","last_undegraded":"2024-09-11T05:51:29.713982+0000","last_fullsized":"2024-09-11T05:51:29.713982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"48'94","reported_seq":579,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:34.079863+0000","last_change":"2024-09-11T05:51:07.507117+0000","last_active":"2024-09-11T05:51:34.079863+0000","last_peered":"2024-09-11T05:51:34.079863+0000","last_clean":"2024-09-11T05:51:34.079863+0000","last_became_active":"2024-09-11T05:51:05.508345+0000","last_became_peered":"2024-09-11T05:51:05.508345+0000","last_unstale":"2024-09-11T05:51:34.079863+0000","last_undegraded":"2024-09-11T05:51:34.079863+0000","last_fullsized":"2024-09-11T05:51:34.079863+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:40.305339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":10279539,"num_objects":21,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":8,"num_read":447,"num_read_kb":2667,"num_write":130,"num_write_kb":6741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"45'70","reported_seq":633,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.792897+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:51:29.792897+0000","last_peered":"2024-09-11T05:51:29.792897+0000","last_clean":"2024-09-11T05:51:29.792897+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:51:29.792897+0000","last_undegraded":"2024-09-11T05:51:29.792897+0000","last_fullsized":"2024-09-11T05:51:29.792897+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:16.405775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":3538107,"num_objects":20,"num_object_clones":9,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":8,"num_read":534,"num_read_kb":1191,"num_write":58,"num_write_kb":1400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"44'138","reported_seq":817,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.691133+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:51:29.691133+0000","last_peered":"2024-09-11T05:51:29.691133+0000","last_clean":"2024-09-11T05:51:29.691133+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:51:29.691133+0000","last_undegraded":"2024-09-11T05:51:29.691133+0000","last_fullsized":"2024-09-11T05:51:29.691133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:00.208140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2649157,"num_objects":10,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":1679,"num_read_kb":2904,"num_write":143,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"47'130","reported_seq":1461,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T05:51:32.812852+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:51:32.812852+0000","last_peered":"2024-09-11T05:51:32.812852+0000","last_clean":"2024-09-11T05:51:32.812852+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:51:32.812852+0000","last_undegraded":"2024-09-11T05:51:32.812852+0000","last_fullsized":"2024-09-11T05:51:32.812852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:41.357825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":24167326,"num_objects":31,"num_object_clones":13,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":11,"num_read":1311,"num_read_kb":9949,"num_write":138,"num_write_kb":5607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"45'59","reported_seq":190,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.799940+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:51:29.799940+0000","last_peered":"2024-09-11T05:51:29.799940+0000","last_clean":"2024-09-11T05:51:29.799940+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:51:29.799940+0000","last_undegraded":"2024-09-11T05:51:29.799940+0000","last_fullsized":"2024-09-11T05:51:29.799940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:57:19.970392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":4979593,"num_objects":15,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":5,"num_read":89,"num_read_kb":1133,"num_write":54,"num_write_kb":557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"45'135","reported_seq":715,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714401+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:51:29.714401+0000","last_peered":"2024-09-11T05:51:29.714401+0000","last_clean":"2024-09-11T05:51:29.714401+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:51:29.714401+0000","last_undegraded":"2024-09-11T05:51:29.714401+0000","last_fullsized":"2024-09-11T05:51:29.714401+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:10.480293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":7991796,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":895,"num_read_kb":6376,"num_write":185,"num_write_kb":2354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"44'106","reported_seq":813,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714191+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:51:29.714191+0000","last_peered":"2024-09-11T05:51:29.714191+0000","last_clean":"2024-09-11T05:51:29.714191+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:51:29.714191+0000","last_undegraded":"2024-09-11T05:51:29.714191+0000","last_fullsized":"2024-09-11T05:51:29.714191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:26.425821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":28311606,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":692,"num_read_kb":1684,"num_write":101,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"44'93","reported_seq":265,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.714372+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:51:29.714372+0000","last_peered":"2024-09-11T05:51:29.714372+0000","last_clean":"2024-09-11T05:51:29.714372+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:51:29.714372+0000","last_undegraded":"2024-09-11T05:51:29.714372+0000","last_fullsized":"2024-09-11T05:51:29.714372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:45:09.436560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":12768584,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":436,"num_read_kb":7115,"num_write":132,"num_write_kb":3749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"45'73","reported_seq":414,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.792596+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:51:29.792596+0000","last_peered":"2024-09-11T05:51:29.792596+0000","last_clean":"2024-09-11T05:51:29.792596+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:51:29.792596+0000","last_undegraded":"2024-09-11T05:51:29.792596+0000","last_fullsized":"2024-09-11T05:51:29.792596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:25:54.732606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12045710,"num_objects":15,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":6,"num_read":314,"num_read_kb":1124,"num_write":66,"num_write_kb":569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"44'101","reported_seq":321,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.713947+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:51:29.713947+0000","last_peered":"2024-09-11T05:51:29.713947+0000","last_clean":"2024-09-11T05:51:29.713947+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:51:29.713947+0000","last_undegraded":"2024-09-11T05:51:29.713947+0000","last_fullsized":"2024-09-11T05:51:29.713947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:26.329003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":9203934,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":5,"num_read":445,"num_read_kb":562,"num_write":96,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"45'372","reported_seq":692,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.793150+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:51:29.793150+0000","last_peered":"2024-09-11T05:51:29.793150+0000","last_clean":"2024-09-11T05:51:29.793150+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:51:29.793150+0000","last_undegraded":"2024-09-11T05:51:29.793150+0000","last_fullsized":"2024-09-11T05:51:29.793150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:14.215102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":5061258,"num_objects":18,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":7,"num_read":1819,"num_read_kb":2805,"num_write":577,"num_write_kb":2180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"44'56","reported_seq":255,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-11T05:51:29.800279+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:51:29.800279+0000","last_peered":"2024-09-11T05:51:29.800279+0000","last_clean":"2024-09-11T05:51:29.800279+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:51:29.800279+0000","last_undegraded":"2024-09-11T05:51:29.800279+0000","last_fullsized":"2024-09-11T05:51:29.800279+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:05:00.278463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":3913381,"num_objects":10,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":150,"num_read_kb":1261,"num_write":55,"num_write_kb":937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":299259486,"num_objects":519,"num_object_clones":201,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":177,"num_read":32276,"num_read_kb":111534,"num_write":5428,"num_write_kb":68578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":55848960,"data_stored":89420931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47348,"kb_used_data":11972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324492,"statfs":{"total":96636764160,"available":96588279808,"internally_reserved":0,"allocated":12259328,"data_stored":18387370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28868,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342972,"statfs":{"total":96636764160,"available":96607203328,"internally_reserved":0,"allocated":1884160,"data_stored":2556789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35804,"kb_used_data":4620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336036,"statfs":{"total":96636764160,"available":96600100864,"internally_reserved":0,"allocated":4730880,"data_stored":7382246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46876,"kb_used_data":15692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324964,"statfs":{"total":96636764160,"available":96588763136,"internally_reserved":0,"allocated":16068608,"data_stored":26457070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42672,"kb_used_data":7328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329168,"statfs":{"total":96636764160,"available":96593068032,"internally_reserved":0,"allocated":7503872,"data_stored":9958823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[30,13],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30624,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341216,"statfs":{"total":96636764160,"available":96605405184,"internally_reserved":0,"allocated":3686400,"data_stored":4918006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36900,"kb_used_data":5716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334940,"statfs":{"total":96636764160,"available":96598978560,"internally_reserved":0,"allocated":5853184,"data_stored":8963459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44788,"kb_used_data":9444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327052,"statfs":{"total":96636764160,"available":96590901248,"internally_reserved":0,"allocated":9670656,"data_stored":14279157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8302592,"data_stored":13204777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5353472,"data_stored":8753398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3190784,"data_stored":4707949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6963200,"data_stored":9721500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15552512,"data_stored":26227307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4177920,"data_stored":7120701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1400832,"data_stored":2355799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10907648,"data_stored":17329500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:51:35.392 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:35.381+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:35.485 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:35.485+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:35.547 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:35.547 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:35.547 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:35.547 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:35.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:35.607 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:35.605+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:35.683 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:35.683 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:35.683 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:35.683 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:35.683 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:35.686 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:35.681+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:35.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:35.728 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:51:35.795 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:35.793+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.854 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20460 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 1015808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.310270+0000", 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5439269319999998, 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020193198999999998, 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20460 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310270+0000", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310270+0000", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310272+0000", 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310280+0000", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1130000000000001e-06 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310294+0000", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4156e-05 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.310305+0000", 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0783e-05 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.330062+0000", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019756942 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.330241+0000", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179536 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.330464+0000", 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022219899999999999 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.858 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20508 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 131072~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5397172349999999, 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020384915, 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20508 2024-09-11T05:51:35.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314481+0000", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.083e-06 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314487+0000", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8030000000000002e-06 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314505+0000", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7779e-05 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314518+0000", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3475999999999999e-05 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334353+0000", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019834766 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334592+0000", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023897499999999999 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334865+0000", 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273033 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20512 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5396610050000001, 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020911447, 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.863 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20512 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314537+0000", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.023e-06 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314544+0000", 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2060000000000004e-06 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314562+0000", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7992e-05 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314574+0000", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2716e-05 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334986+0000", 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020411838000000002 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335179+0000", 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019315000000000001 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335448+0000", 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026852199999999999 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20515 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 409600~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.315795+0000", 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.538402338, 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020250338999999999, 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20515 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315795+0000", 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315795+0000", 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315797+0000", 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2239999999999998e-06 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315809+0000", 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1962e-05 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315841+0000", 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1321e-05 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315862+0000", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1702999999999999e-05 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335561+0000", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019699195999999999 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335784+0000", 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022272899999999999 2024-09-11T05:51:35.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336045+0000", 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026120400000000002 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20522 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 524288~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.538295111, 2024-09-11T05:51:35.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020763599000000001, 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20522 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315904+0000", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315911+0000", 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7530000000000001e-06 2024-09-11T05:51:35.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315933+0000", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1868000000000001e-05 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315949+0000", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6146000000000001e-05 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336229+0000", 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020279993999999999 2024-09-11T05:51:35.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336424+0000", 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019419000000000001 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336666+0000", 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024231799999999999 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20524 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5382251089999999, 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021361196999999998, 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20524 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315974+0000", 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.367e-06 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315981+0000", 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3690000000000001e-06 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316034+0000", 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2694000000000001e-05 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316053+0000", 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9324999999999999e-05 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336785+0000", 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020731611 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337045+0000", 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000260287 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337333+0000", 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028854400000000001 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20528 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 802816~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5380619920000003, 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021755004000000001, 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20528 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316137+0000", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6929999999999999e-06 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316144+0000", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3889999999999999e-06 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316166+0000", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1685999999999999e-05 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316183+0000", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6509000000000001e-05 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337439+0000", 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021256744000000001 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337660+0000", 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022058000000000001 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337890+0000", 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023040300000000001 2024-09-11T05:51:35.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20529 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 917504~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5379912500000001, 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022138649999999999, 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20529 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316208+0000", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4899999999999999e-06 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316220+0000", 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2629000000000001e-05 2024-09-11T05:51:35.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316240+0000", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9664999999999999e-05 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316255+0000", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929000000000001e-05 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337958+0000", 2024-09-11T05:51:35.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021703009999999998 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338131+0000", 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172991 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338345+0000", 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021393599999999999 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20532 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5379225879999998, 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022552875, 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20532 2024-09-11T05:51:35.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316276+0000", 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.04e-06 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316282+0000", 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7869999999999998e-06 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316300+0000", 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7965000000000002e-05 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316314+0000", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4382999999999999e-05 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338416+0000", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022102087999999999 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338602+0000", 2024-09-11T05:51:35.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000186212 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338828+0000", 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0002254 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23877 3.7 3:e4bc48c1:::rbd_data.11244d01bf21.000000000000002f:f [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1676966790000001, 2024-09-11T05:51:35.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022057406000000002, 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23877 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4400000000000001e-07 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686503+0000", 2024-09-11T05:51:35.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1490000000000001e-06 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686510+0000", 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3419999999999996e-06 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686514+0000", 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7350000000000004e-06 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.702863+0000", 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016348260999999999 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.702934+0000", 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1564999999999997e-05 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.708558+0000", 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056239100000000002 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.899 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23912 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1674085930000002, 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020603792999999999, 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.900 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23912 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3000000000000002e-07 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686791+0000", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.863e-06 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686796+0000", 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4249999999999997e-06 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686801+0000", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2379999999999999e-06 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.706863+0000", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020062582999999998 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.706936+0000", 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3106000000000001e-05 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707393+0000", 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045624800000000003 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23913 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 917504~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.167390267, 2024-09-11T05:51:35.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020767105000000001, 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23913 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:35.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686808+0000", 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8800000000000002e-07 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686810+0000", 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6450000000000001e-06 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686817+0000", 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686822+0000", 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6739999999999996e-06 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707416+0000", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020593923 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707492+0000", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6050000000000005e-05 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707574+0000", 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2163999999999995e-05 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.908 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24364 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 278528~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1561612600000002, 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021347893999999999, 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.909 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24364 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698037+0000", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8100000000000003e-07 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698039+0000", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2680000000000001e-06 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698045+0000", 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1419999999999998e-06 2024-09-11T05:51:35.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698050+0000", 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7469999999999996e-06 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.714965+0000", 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016914769 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.717633+0000", 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026687899999999999 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719384+0000", 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001750597 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24365 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 352256~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.913 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1561413310000002, 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021530918999999999, 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24365 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6199999999999999e-07 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698058+0000", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0320000000000002e-06 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698064+0000", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0090000000000004e-06 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698069+0000", 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7389999999999999e-06 2024-09-11T05:51:35.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719418+0000", 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021349283 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719501+0000", 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2282999999999998e-05 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719587+0000", 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6211000000000005e-05 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24366 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1561219569999999, 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021705247, 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.918 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24366 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698076+0000", 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8299999999999997e-07 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698078+0000", 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0379999999999998e-06 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698084+0000", 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.913e-06 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698089+0000", 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6820000000000002e-06 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719617+0000", 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021528623 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719696+0000", 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8755000000000001e-05 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719781+0000", 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4653000000000004e-05 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.922 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24367 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1561028069999999, 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021876105999999999, 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24367 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:51:35.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698097+0000", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9080000000000002e-06 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698103+0000", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.879e-06 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698107+0000", 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7199999999999997e-06 2024-09-11T05:51:35.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719811+0000", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021703940000000001 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719887+0000", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6081000000000006e-05 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719971+0000", 2024-09-11T05:51:35.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3208999999999996e-05 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24368 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 704512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1560838709999999, 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022050626, 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.928 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24368 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7300000000000001e-07 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698116+0000", 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3910000000000001e-06 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698122+0000", 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698127+0000", 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720001+0000", 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021873373000000002 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720078+0000", 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7645000000000006e-05 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720164+0000", 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5667999999999994e-05 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24369 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1560623570000002, 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023716779, 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24369 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.933 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698136+0000", 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698138+0000", 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3980000000000002e-06 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698145+0000", 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8220000000000001e-06 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698150+0000", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9830000000000004e-06 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720194+0000", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022044587000000001 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720270+0000", 2024-09-11T05:51:35.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6077999999999999e-05 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721852+0000", 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001581442 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24371 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 851968~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1560191780000002, 2024-09-11T05:51:35.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023858969000000001, 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24371 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698179+0000", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698181+0000", 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9910000000000001e-06 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698186+0000", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7520000000000002e-06 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698191+0000", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2939999999999999e-06 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721888+0000", 2024-09-11T05:51:35.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023697685999999999 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721965+0000", 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7156000000000005e-05 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722037+0000", 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1717999999999993e-05 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.941 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24372 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.1560005979999999, 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023933343999999999, 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:35.942 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24372 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5999999999999999e-07 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698203+0000", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2920000000000001e-06 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698210+0000", 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6800000000000004e-06 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698215+0000", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6519999999999997e-06 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722055+0000", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023840296 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722096+0000", 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0775000000000002e-05 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:35.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722130+0000", 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4289000000000003e-05 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:35.946 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:35.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:35.948 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:35.945+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:36.049 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:36.049+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:36.084 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:36.084 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:36.084 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:36.084 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:36.085 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-09-11T05:51:36.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:36.149 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:36.145+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:36.186 INFO:teuthology.orchestra.run.smithi165.stdout:2546 read 0x726c2c3 thru 0x727553c (0x927a bytes) 2024-09-11T05:51:36.188 INFO:teuthology.orchestra.run.smithi165.stdout:2547 write 0xdb5442f thru 0xdb61685 (0xd257 bytes) 2024-09-11T05:51:36.201 INFO:teuthology.orchestra.run.smithi165.stdout:2548 writesame 0x8d4bb79 thru 0x8d57673 data_size 0x1(0xbafb bytes) 2024-09-11T05:51:36.245 INFO:teuthology.orchestra.run.smithi165.stdout:2550 trunc from 0xe0c0593 to 0x5647a90 2024-09-11T05:51:36.251 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:36.249+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:36.272 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:36.272 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:36.272 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:36.272 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:36.273 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:36.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:36.315 INFO:teuthology.orchestra.run.smithi165.stdout:2551 write 0xebe3ef7 thru 0xebe77f5 (0x38ff bytes) 2024-09-11T05:51:36.322 INFO:teuthology.orchestra.run.smithi165.stdout:2554 write 0x1a3008a thru 0x1a3bd47 (0xbcbe bytes) 2024-09-11T05:51:36.336 INFO:teuthology.orchestra.run.smithi165.stdout:2559 write 0x9880a42 thru 0x988f51d (0xeadc bytes) 2024-09-11T05:51:36.344 INFO:teuthology.orchestra.run.smithi165.stdout:2560 read 0xd1db518 thru 0xd1e7e16 (0xc8ff bytes) 2024-09-11T05:51:36.345 INFO:teuthology.orchestra.run.smithi165.stdout:2561 write 0xe968db1 thru 0xe96c889 (0x3ad9 bytes) 2024-09-11T05:51:36.350 INFO:teuthology.orchestra.run.smithi165.stdout:2562 punch from 0x11ac464 to 0x11b8c04, (0xc7a0 bytes) 2024-09-11T05:51:36.350 INFO:teuthology.orchestra.run.smithi165.stdout:2563 read 0x8d648bb thru 0x8d7042d (0xbb73 bytes) 2024-09-11T05:51:36.350 INFO:teuthology.orchestra.run.smithi165.stdout:2564 read 0x6a5bba7 thru 0x6a62253 (0x66ad bytes) 2024-09-11T05:51:36.350 INFO:teuthology.orchestra.run.smithi165.stdout:2565 writesame 0x268d505 thru 0x2698702 data_size 0x1(0xb1fe bytes) 2024-09-11T05:51:36.351 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:36.349+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:36.385 INFO:teuthology.orchestra.run.smithi165.stdout:2568 punch from 0xcd19542 to 0xcd1cf5b, (0x3a19 bytes) 2024-09-11T05:51:36.385 INFO:teuthology.orchestra.run.smithi165.stdout:2569 writesame 0x6e7bc5d thru 0x6e8325a data_size 0xb(0x75fe bytes) 2024-09-11T05:51:36.407 INFO:teuthology.orchestra.run.smithi165.stdout:2570 write 0xec077b5 thru 0xec08913 (0x115f bytes) 2024-09-11T05:51:36.416 INFO:teuthology.orchestra.run.smithi165.stdout:2571 read 0x3c8cdbd thru 0x3c90b43 (0x3d87 bytes) 2024-09-11T05:51:36.417 INFO:teuthology.orchestra.run.smithi165.stdout:2572 write 0xcfe175f thru 0xcfece4c (0xb6ee bytes) 2024-09-11T05:51:36.426 INFO:teuthology.orchestra.run.smithi165.stdout:2574 writesame 0xe3beda7 thru 0xe3c2e50 data_size 0x3e(0x40aa bytes) 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:11636 3.1a 3:5ee2e31d:::rbd_data.1124518b2c12.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.210576912000001, 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023307497, 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11636 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217164+0000", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.217167+0000", 2024-09-11T05:51:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0000000000000001e-06 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221731+0000", 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045638420000000002 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221766+0000", 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5410000000000001e-05 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221782+0000", 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5899000000000001e-05 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.221933+0000", 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015070599999999999 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.222312+0000", 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037949700000000002 2024-09-11T05:51:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.224398+0000", 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00208618 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.237521+0000", 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013122642 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240180+0000", 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002659509 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240337+0000", 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156269 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.240471+0000", 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013454299999999999 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.434 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19097 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 278528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3381684400000005, 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022586779000000001, 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19097 2024-09-11T05:51:36.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089572+0000", 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089573+0000", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1370000000000001e-06 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089579+0000", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3249999999999998e-06 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089593+0000", 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4175999999999999e-05 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.089602+0000", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9320000000000002e-06 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111635+0000", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022033437999999999 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.111872+0000", 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023689199999999999 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112159+0000", 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028687899999999998 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19098 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 442368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3374452809999999, 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022463285999999999, 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19098 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090295+0000", 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090297+0000", 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090304+0000", 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2830000000000004e-06 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090327+0000", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3264e-05 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090338+0000", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0285000000000001e-05 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112249+0000", 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021911336 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112482+0000", 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233005 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112759+0000", 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000276483 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.443 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19099 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 606208~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3372399789999996, 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022863720000000001, 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19099 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.444 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090501+0000", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090502+0000", 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1829999999999999e-06 2024-09-11T05:51:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090509+0000", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9229999999999996e-06 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090525+0000", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6551000000000001e-05 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090535+0000", 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0071999999999999e-05 2024-09-11T05:51:36.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.112846+0000", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022310399000000002 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113083+0000", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000237445 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113364+0000", 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028114699999999998 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.447 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19100 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3371888649999999, 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023303653000000001, 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19100 2024-09-11T05:51:36.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090552+0000", 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4700000000000001e-07 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090556+0000", 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4429999999999999e-06 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090565+0000", 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.302e-06 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090572+0000", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.233e-06 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113495+0000", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022922464 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113656+0000", 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016068899999999999 2024-09-11T05:51:36.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113855+0000", 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019987499999999999 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:19105 3.9 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:c [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3371550120000002, 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023901565999999999, 2024-09-11T05:51:36.452 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19105 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090586+0000", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5300000000000004e-07 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090589+0000", 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.19e-06 2024-09-11T05:51:36.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090598+0000", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8659999999999995e-06 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.090605+0000", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.113961+0000", 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023356247 2024-09-11T05:51:36.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.114202+0000", 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000240261 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.114487+0000", 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028545300000000002 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.456 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:22255 3.9 3:97eabedf:::rbd_data.11244d01bf21.0000000000000021:f [read 737280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.609155+0000", 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.8185857800000003, 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024838948999999999, 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 22255 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609155+0000", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609155+0000", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609155+0000", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.53e-07 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609157+0000", 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0660000000000002e-06 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609163+0000", 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0610000000000001e-06 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.609168+0000", 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7219999999999999e-06 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620555+0000", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01138745 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.620632+0000", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6941999999999996e-05 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.633994+0000", 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013361355 2024-09-11T05:51:36.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29046 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121716+0000", 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30602487, 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022290270000000001, 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.461 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29046 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121716+0000", 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121716+0000", 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121716+0000", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4000000000000002e-07 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121719+0000", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.774e-06 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121729+0000", 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5689999999999994e-06 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121736+0000", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8950000000000001e-06 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.143999+0000", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022263451 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144006+0000", 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.041e-06 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.464 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29050 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121744+0000", 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30599635200000003, 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022313873000000001, 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.465 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29050 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121744+0000", 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121744+0000", 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121745+0000", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8999999999999996e-07 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121750+0000", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6879999999999998e-06 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121758+0000", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4829999999999999e-06 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121765+0000", 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2309999999999999e-06 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144042+0000", 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022276562999999999 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144058+0000", 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6218e-05 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29061 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121774+0000", 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30596642899999998, 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022307286999999999, 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29061 2024-09-11T05:51:36.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121774+0000", 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121774+0000", 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121775+0000", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4700000000000001e-07 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121777+0000", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.638e-06 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121816+0000", 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8661000000000001e-05 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121824+0000", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4749999999999993e-06 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144075+0000", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022250215 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144081+0000", 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7510000000000002e-06 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29067 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121862+0000", 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30587885199999998, 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022291044999999999, 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29067 2024-09-11T05:51:36.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121862+0000", 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121862+0000", 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121862+0000", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.27e-07 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121864+0000", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0959999999999999e-06 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121870+0000", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7740000000000001e-06 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121874+0000", 2024-09-11T05:51:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8820000000000003e-06 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144146+0000", 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02227165 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144153+0000", 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.216e-06 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29069 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121879+0000", 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30586167199999997, 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022321705000000001, 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29069 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121879+0000", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121879+0000", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121879+0000", 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4299999999999999e-07 2024-09-11T05:51:36.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121886+0000", 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0079999999999996e-06 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121891+0000", 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7249999999999997e-06 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121896+0000", 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5700000000000003e-06 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144188+0000", 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022292811999999999 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144201+0000", 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2247e-05 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29073 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121901+0000", 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30583995800000002, 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022338622999999998, 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29073 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.481 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121901+0000", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121901+0000", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121901+0000", 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3299999999999998e-07 2024-09-11T05:51:36.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121903+0000", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5710000000000001e-06 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121907+0000", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3900000000000003e-06 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121910+0000", 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5659999999999998e-06 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144231+0000", 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022320215000000001 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144239+0000", 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5480000000000002e-06 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29075 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121915+0000", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30582527799999998, 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022375691999999999, 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29075 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121915+0000", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121915+0000", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121916+0000", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121917+0000", 2024-09-11T05:51:36.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5549999999999999e-06 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121921+0000", 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2649999999999998e-06 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121925+0000", 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9149999999999998e-06 2024-09-11T05:51:36.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144281+0000", 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022355562999999998 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144291+0000", 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0056e-05 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29079 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121960+0000", 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30578089400000003, 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022371502000000001, 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29079 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121960+0000", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121960+0000", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121960+0000", 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.63e-07 2024-09-11T05:51:36.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121962+0000", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.069e-06 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121968+0000", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6250000000000004e-06 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121972+0000", 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2139999999999998e-06 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144324+0000", 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022352130000000001 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144331+0000", 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0010000000000004e-06 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.492 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29082 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121978+0000", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30576248099999997, 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022396124999999999, 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29082 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121978+0000", 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121978+0000", 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121978+0000", 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9900000000000002e-07 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121980+0000", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5239999999999999e-06 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121984+0000", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3309999999999997e-06 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121988+0000", 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519999999999999e-06 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144365+0000", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022376627999999999 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144374+0000", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6909999999999992e-06 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29088 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.121992+0000", 2024-09-11T05:51:36.496 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30574815500000002, 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022426135, 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29088 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121992+0000", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121992+0000", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121993+0000", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0400000000000002e-07 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.121995+0000", 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0770000000000002e-06 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122002+0000", 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4139999999999997e-06 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122009+0000", 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5549999999999999e-06 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144412+0000", 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022402756999999999 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144419+0000", 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0280000000000003e-06 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29090 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.500 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.122017+0000", 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30572353699999999, 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02245279, 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29090 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122017+0000", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122017+0000", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122018+0000", 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5400000000000002e-07 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122020+0000", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9749999999999999e-06 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122029+0000", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1810000000000001e-06 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122035+0000", 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8249999999999999e-06 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144460+0000", 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022424825999999998 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144470+0000", 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5289999999999997e-06 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29095 3.9 3.1873fae9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.122045+0000", 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30569587100000001, 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022464284000000001, 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29095 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.505 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122045+0000", 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122045+0000", 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122045+0000", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122047+0000", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9850000000000001e-06 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122052+0000", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8709999999999996e-06 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.122056+0000", 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9929999999999997e-06 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144501+0000", 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022445104 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.144509+0000", 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9570000000000002e-06 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:29539 3.9 3:9653dd27:::rbd_data.1124b2f65e38.0000000000000026:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.365055+0000", 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.062685828999999998, 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022919097999999999, 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29539 2024-09-11T05:51:36.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365055+0000", 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365055+0000", 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:36.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365056+0000", 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0690000000000001e-06 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365062+0000", 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.733e-06 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365074+0000", 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2879e-05 2024-09-11T05:51:36.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365080+0000", 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6790000000000002e-06 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365139+0000", 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8770000000000001e-05 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.365272+0000", 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013318600000000001 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.369778+0000", 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045062180000000002 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.387826+0000", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018047236000000001 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.387883+0000", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7009000000000003e-05 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.387923+0000", 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0821000000000001e-05 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:36.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.387974+0000", 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0498000000000003e-05 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:36.515 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:36.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:36.517 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:36.517+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:36.517 INFO:teuthology.orchestra.run.smithi165.stdout:2575 write 0x17dc16a thru 0x17e6765 (0xa5fc bytes) 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:2577 write 0x15d48ae thru 0x15d5488 (0xbdb bytes) 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:2578 trunc from 0xec08914 to 0xdbe9437 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:2579 writesame 0x25b1b0c thru 0x25b7dcf data_size 0x81(0x62c4 bytes) 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:2580 write 0xad9a1e9 thru 0xad9dcf6 (0x3b0e bytes) 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:2581 writesame 0xd6eb2d0 thru 0xd6ec5f6 data_size 0x1(0x1327 bytes) 2024-09-11T05:51:36.518 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:36.523 INFO:teuthology.orchestra.run.smithi165.stdout:2583 trunc from 0xdbe9437 to 0x9704ec1 2024-09-11T05:51:36.549 INFO:teuthology.orchestra.run.smithi165.stdout:2584 write 0x3b0570c thru 0x3b1047d (0xad72 bytes) 2024-09-11T05:51:36.569 INFO:teuthology.orchestra.run.smithi165.stdout:2586 write 0x6bdd26b thru 0x6be00e8 (0x2e7e bytes) 2024-09-11T05:51:36.578 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:36.599 INFO:teuthology.orchestra.run.smithi165.stdout:2588 writesame 0xcdea268 thru 0xcdeca1c data_size 0x13(0x27b5 bytes) 2024-09-11T05:51:36.599 INFO:teuthology.orchestra.run.smithi165.stdout:2589 write 0x954cdf6 thru 0x9554d74 (0x7f7f bytes) 2024-09-11T05:51:36.607 INFO:teuthology.orchestra.run.smithi165.stdout:2590 writesame 0x5fb0438 thru 0x5fb2499 data_size 0x5(0x2062 bytes) 2024-09-11T05:51:36.613 INFO:teuthology.orchestra.run.smithi165.stdout:2591 write 0x22a3331 thru 0x22b0ad2 (0xd7a2 bytes) 2024-09-11T05:51:36.618 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:36.613+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:36.634 INFO:teuthology.orchestra.run.smithi165.stdout:2594 read 0x842ca97 thru 0x843781c (0xad86 bytes) 2024-09-11T05:51:36.634 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:36.634 INFO:teuthology.orchestra.run.smithi165.stdout:2597 write 0xae52f05 thru 0xae54c81 (0x1d7d bytes) 2024-09-11T05:51:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:2600 read 0x6954241 thru 0x695b76e (0x752e bytes) 2024-09-11T05:51:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:2601 read 0xc4e4ac9 thru 0xc4f2f50 (0xe488 bytes) 2024-09-11T05:51:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:2602 clone 15 order 19 su 2048 sc 15 2024-09-11T05:51:36.695 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:36.695 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:36.695 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:36.695 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:36.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:36.752 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:36.753+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:36.822 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:36.821+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:36.842 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:36.842 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:36.842 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:36.842 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:36.842 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:36.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:36.871 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone14 from 0xcdeca1d (overlap 0x5647a90) to 0x2b23d48 2024-09-11T05:51:36.944 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:36.945+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:37.024 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:37.021+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.292064328, 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:37.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12821 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.070696036999999, 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028055824, 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12821 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979063+0000", 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979066+0000", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8839999999999998e-06 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979079+0000", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3159999999999999e-05 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979111+0000", 2024-09-11T05:51:37.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2406000000000001e-05 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979128+0000", 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6733000000000001e-05 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979265+0000", 2024-09-11T05:51:37.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013638600000000001 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.979400+0000", 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135752 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.983348+0000", 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039480169999999998 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.006806+0000", 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023457281999999999 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.006917+0000", 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111462 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.007011+0000", 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3968999999999994e-05 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:23.007119+0000", 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107773 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:21848 3.1d 3:bd6be448:::rbd_data.11244d01bf21.0000000000000023:f [read 868352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.4494043730000001, 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025560946000000001, 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 21848 2024-09-11T05:51:37.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600355+0000", 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4400000000000001e-07 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600357+0000", 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9300000000000002e-06 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600363+0000", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4489999999999996e-06 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.600368+0000", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.977e-06 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618805+0000", 2024-09-11T05:51:37.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018436113000000001 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.618884+0000", 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9640000000000003e-05 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.625916+0000", 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070313930000000004 2024-09-11T05:51:37.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.2488362149999999, 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.071 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26725 3.c 3:30e4d020:::rbd_data.1124a9e466a.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12058624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.808492+0000", 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.2412671319999999, 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027967097, 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.078 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26725 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808492+0000", 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808492+0000", 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808494+0000", 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.694e-06 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816204+0000", 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077102969999999996 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816225+0000", 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1455e-05 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816236+0000", 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.042e-05 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816321+0000", 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5383000000000006e-05 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816632+0000", 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031089799999999999 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.818282+0000", 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001649838 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.834673+0000", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016391288 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.836326+0000", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016531759999999999 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.836396+0000", 2024-09-11T05:51:37.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9731e-05 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.836459+0000", 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2916999999999994e-05 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26816 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12058624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.816272+0000", 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.2334874029999998, 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028719109, 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26816 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816272+0000", 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816272+0000", 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.816274+0000", 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9489999999999999e-06 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.825232+0000", 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089582919999999996 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.825253+0000", 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1542000000000001e-05 2024-09-11T05:51:37.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.825265+0000", 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.208e-05 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.825357+0000", 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1533000000000006e-05 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.825543+0000", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018603399999999999 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.826677+0000", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011335830000000001 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.844581+0000", 2024-09-11T05:51:37.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01790404 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.844926+0000", 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034516700000000001 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.844962+0000", 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6174999999999998e-05 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.844991+0000", 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8714e-05 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28476 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 819200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.108370+0000", 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.94138897499999996, 2024-09-11T05:51:37.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022022984999999998, 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28476 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108370+0000", 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108370+0000", 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108371+0000", 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2929999999999999e-06 2024-09-11T05:51:37.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108378+0000", 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108397+0000", 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9284999999999999e-05 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108411+0000", 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4469e-05 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130232+0000", 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021820809 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130317+0000", 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5006000000000006e-05 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130393+0000", 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5933000000000006e-05 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28477 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 909312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.108431+0000", 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.94132848999999996, 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022194662, 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28477 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108431+0000", 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108431+0000", 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108432+0000", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.28e-06 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108437+0000", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5219999999999998e-06 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108456+0000", 2024-09-11T05:51:37.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8355999999999998e-05 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108471+0000", 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5194999999999999e-05 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130436+0000", 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021965612999999998 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130531+0000", 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4598000000000005e-05 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130625+0000", 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4098000000000006e-05 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28478 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 1007616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.108492+0000", 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.94126725300000003, 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022340854, 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.102 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28478 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108492+0000", 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108492+0000", 2024-09-11T05:51:37.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108493+0000", 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.297e-06 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108499+0000", 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9000000000000003e-06 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108518+0000", 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8800999999999998e-05 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.108532+0000", 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4358999999999999e-05 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130682+0000", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022150306000000002 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130761+0000", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8994999999999998e-05 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.130833+0000", 2024-09-11T05:51:37.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1196000000000001e-05 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28542 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 966656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.111883+0000", 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93787569100000001, 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021941728000000001, 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28542 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111883+0000", 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111883+0000", 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111884+0000", 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.54e-07 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111887+0000", 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111895+0000", 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549999999999995e-06 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.111902+0000", 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4659999999999996e-06 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.133539+0000", 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021637780999999998 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.133701+0000", 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016182499999999999 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.133825+0000", 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123779 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28674 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 155648~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116019+0000", 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93374016299999996, 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022212237999999999, 2024-09-11T05:51:37.112 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28674 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116019+0000", 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116019+0000", 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116019+0000", 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9299999999999998e-07 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116022+0000", 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.621e-06 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116031+0000", 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5629999999999992e-06 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116037+0000", 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6769999999999998e-06 2024-09-11T05:51:37.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.137920+0000", 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021882902999999999 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138023+0000", 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103282 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138231+0000", 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020759899999999999 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28677 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 245760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116048+0000", 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93371146299999996, 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022383307000000002, 2024-09-11T05:51:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28677 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116048+0000", 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116048+0000", 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116048+0000", 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4899999999999995e-07 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116051+0000", 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116059+0000", 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6719999999999993e-06 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116067+0000", 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1400000000000002e-06 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138292+0000", 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022225458999999999 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138353+0000", 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0513000000000002e-05 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138431+0000", 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8250000000000005e-05 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28679 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 344064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116075+0000", 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93368409100000005, 2024-09-11T05:51:37.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022722145999999999, 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28679 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116075+0000", 2024-09-11T05:51:37.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116075+0000", 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116076+0000", 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.71e-07 2024-09-11T05:51:37.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116079+0000", 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7709999999999998e-06 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116087+0000", 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7150000000000004e-06 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116094+0000", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4239999999999998e-06 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138483+0000", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022389167000000001 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138580+0000", 2024-09-11T05:51:37.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6967999999999997e-05 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138797+0000", 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021723000000000001 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28682 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 442368~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116104+0000", 2024-09-11T05:51:37.127 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93365503699999997, 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0228376, 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28682 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116104+0000", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116104+0000", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116105+0000", 2024-09-11T05:51:37.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6599999999999996e-07 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116115+0000", 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0189e-05 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116123+0000", 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5520000000000005e-06 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116130+0000", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7240000000000003e-06 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138818+0000", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022687766000000002 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138873+0000", 2024-09-11T05:51:37.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5618999999999997e-05 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.138942+0000", 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8083999999999994e-05 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28685 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 540672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116140+0000", 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.933619423, 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023182390000000001, 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28685 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116140+0000", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116140+0000", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116140+0000", 2024-09-11T05:51:37.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1399999999999997e-07 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116143+0000", 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5239999999999999e-06 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116151+0000", 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2300000000000008e-06 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116158+0000", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8510000000000001e-06 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139011+0000", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022853459999999999 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139148+0000", 2024-09-11T05:51:37.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013679499999999999 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139322+0000", 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017391600000000001 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28688 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 630784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116166+0000", 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93359270999999999, 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023291293000000001, 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28688 2024-09-11T05:51:37.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116166+0000", 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116166+0000", 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116167+0000", 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7999999999999995e-07 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116169+0000", 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6129999999999998e-06 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116179+0000", 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1479999999999995e-06 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116185+0000", 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6429999999999998e-06 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139346+0000", 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023161004999999998 2024-09-11T05:51:37.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139402+0000", 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5686000000000002e-05 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139458+0000", 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5618000000000002e-05 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28690 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116194+0000", 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93356511799999997, 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023571377000000001, 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28690 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116194+0000", 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116194+0000", 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116194+0000", 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6899999999999997e-07 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116197+0000", 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.824e-06 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116206+0000", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3690000000000007e-06 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116212+0000", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4509999999999997e-06 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139528+0000", 2024-09-11T05:51:37.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023316083000000001 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139627+0000", 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8356999999999994e-05 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139765+0000", 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013872400000000001 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.147 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28693 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 827392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116220+0000", 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93353887499999999, 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023703157999999998, 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.148 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28693 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116220+0000", 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116220+0000", 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116221+0000", 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116223+0000", 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6199999999999999e-06 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116232+0000", 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5320000000000006e-06 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116239+0000", 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7789999999999998e-06 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139814+0000", 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023575858000000002 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139867+0000", 2024-09-11T05:51:37.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2834000000000003e-05 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139923+0000", 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5995999999999997e-05 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28696 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 925696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.153 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116247+0000", 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93351235600000004, 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024010863, 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28696 2024-09-11T05:51:37.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116247+0000", 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116247+0000", 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116247+0000", 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1800000000000005e-07 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116250+0000", 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4820000000000001e-06 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116258+0000", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4509999999999992e-06 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116265+0000", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4620000000000001e-06 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.139991+0000", 2024-09-11T05:51:37.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023726189000000002 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.140103+0000", 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112392 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.140257+0000", 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015416900000000001 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:28699 3.16 3:6d58c0e3:::rbd_data.112453064ca6.0000000000000004:e [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:36.116276+0000", 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.93348348199999998, 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024119613000000002, 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:37.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28699 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116276+0000", 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116276+0000", 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116276+0000", 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116279+0000", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.463e-06 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116287+0000", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4829999999999999e-06 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.116294+0000", 2024-09-11T05:51:37.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.140290+0000", 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023996099999999999 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.140343+0000", 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2512999999999997e-05 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:36.140395+0000", 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2414999999999999e-05 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.164 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:37.165 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-09-11T05:51:37.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:37.166 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:37.161+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:37.267 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:37.261+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:37.337 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:37.337 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:37.337 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:37.337 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:37.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:37.367 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:37.365+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:37.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:37.468 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:37.465+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:37.471 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:37.471 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:37.471 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:37.471 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:37.471 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:37.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:37.510 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:37.513 INFO:teuthology.orchestra.run.smithi165.stdout:2605 trunc from 0xcdeca1d to 0xb0b64cc 2024-09-11T05:51:37.549 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:37.549 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:37.572 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:37.572 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:37.572 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:37.573+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:37.594 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:37.594 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:37.609 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:37.612 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:37.612 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:37.618 INFO:teuthology.orchestra.run.smithi165.stdout:2607 trunc from 0xb0b64cc to 0x5269d37 2024-09-11T05:51:37.634 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:37.634 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:37.673 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:37.673 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:37.674 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:37.673+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:1026 3.a e30/28 3:50a9d2c1:::rbd_data.112475afb12f.000000000000004c:head v 30'2, mlcod=0'0)", 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.837715532000001, 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043959489999999997, 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851063+0000", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899999999999999e-06 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851141+0000", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851182+0000", 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1207e-05 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851204+0000", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1812999999999999e-05 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851341+0000", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013705600000000001 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851391+0000", 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9811999999999997e-05 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855349+0000", 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00395825 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855456+0000", 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107582 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6402 3.a e33/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 33'21, mlcod=33'19)", 2024-09-11T05:51:37.693 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.807509759999999, 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003674802, 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:37.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881269+0000", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6539999999999999e-06 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881325+0000", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609000000000002e-05 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881385+0000", 2024-09-11T05:51:37.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0043000000000001e-05 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881406+0000", 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1670000000000001e-05 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881582+0000", 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017577200000000001 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881649+0000", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6608999999999999e-05 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884821+0000", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003172336 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884941+0000", 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120109 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6610 3.10 e33/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 33'15, mlcod=33'13)", 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.218921+0000", 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.469855369000001, 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025173040000000002, 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.218921+0000", 2024-09-11T05:51:37.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.218921+0000", 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.218923+0000", 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2699999999999999e-06 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.218939+0000", 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6067e-05 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.218981+0000", 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.193e-05 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219001+0000", 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0106e-05 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219366+0000", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000364791 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219420+0000", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4388999999999997e-05 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.221320+0000", 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018996219999999999 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.221438+0000", 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011812899999999999 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6616 3.1e e33/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 33'22, mlcod=33'20)", 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.46964084, 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032823980000000002, 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219136+0000", 2024-09-11T05:51:37.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.296e-06 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219148+0000", 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1257000000000001e-05 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219171+0000", 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3056000000000001e-05 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219186+0000", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5684000000000001e-05 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220037+0000", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085078300000000003 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220093+0000", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5526000000000003e-05 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222304+0000", 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022115070000000001 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222418+0000", 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011328900000000001 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6863 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'19, mlcod=33'17)", 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.613114+0000", 2024-09-11T05:51:37.706 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.075662054999999, 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025277440000000002, 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613114+0000", 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613114+0000", 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613115+0000", 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9999999999999996e-07 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613176+0000", 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1509999999999999e-05 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613198+0000", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2076e-05 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613218+0000", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9766999999999999e-05 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613389+0000", 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017104500000000001 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.613452+0000", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2362999999999999e-05 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.615528+0000", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020767709999999998 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.615642+0000", 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113412 2024-09-11T05:51:37.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6938 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'20, mlcod=33'18)", 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.720765+0000", 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.968010626000002, 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002520361, 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720765+0000", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720765+0000", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720768+0000", 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720828+0000", 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0152999999999998e-05 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720871+0000", 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3220000000000003e-05 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.720896+0000", 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5043000000000001e-05 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.721064+0000", 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016829200000000001 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.721125+0000", 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0282999999999999e-05 2024-09-11T05:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.723186+0000", 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002061701 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.723286+0000", 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9549000000000005e-05 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.715 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7072 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'24, mlcod=34'22)", 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.757691432000001, 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033394750000000002, 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931087+0000", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931105+0000", 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8652999999999999e-05 2024-09-11T05:51:37.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931148+0000", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2355999999999999e-05 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931167+0000", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.929e-05 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931387+0000", 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000220225 2024-09-11T05:51:37.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931441+0000", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3808999999999999e-05 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934348+0000", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029071850000000001 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934424+0000", 2024-09-11T05:51:37.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5887e-05 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.695514547999998, 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:51:37.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7161 3.1e e34/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 34'27, mlcod=34'25)", 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.616341051999999, 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044832179999999998, 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072438+0000", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1599999999999998e-06 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072465+0000", 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6855999999999998e-05 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072505+0000", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0323e-05 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072527+0000", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2016000000000001e-05 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072689+0000", 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161943 2024-09-11T05:51:37.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072751+0000", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1241999999999994e-05 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076795+0000", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040443759999999997 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076918+0000", 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012330199999999999 2024-09-11T05:51:37.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:11928 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'62, mlcod=36'60)", 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.083930691999999, 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037441639999999999, 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604846+0000", 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1230000000000001e-06 2024-09-11T05:51:37.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604861+0000", 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4877e-05 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604892+0000", 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0973000000000003e-05 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604915+0000", 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2796999999999999e-05 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605079+0000", 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016418800000000001 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605138+0000", 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8579999999999998e-05 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608482+0000", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033439770000000001 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608590+0000", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107649 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:12634 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'87, mlcod=37'85)", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:37.733 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.168469764999999, 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035953420000000001, 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:37.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520308+0000", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8339999999999999e-06 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520321+0000", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2523e-05 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520351+0000", 2024-09-11T05:51:37.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0580000000000002e-05 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520366+0000", 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4866e-05 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520499+0000", 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132636 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520540+0000", 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0883000000000001e-05 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523805+0000", 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032651749999999999 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523902+0000", 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6844999999999999e-05 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15016 3.10 e41/28 3:0f623401:::rbd_data.11243bbd48bb.0000000000000009:head v 41'52, mlcod=39'49)", 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.873083318999999, 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028038719999999998, 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:37.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815694+0000", 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1200000000000001e-06 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815958+0000", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026459200000000002 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815992+0000", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3983000000000003e-05 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816009+0000", 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.645e-05 2024-09-11T05:51:37.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816125+0000", 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116152 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816166+0000", 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0803000000000002e-05 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818356+0000", 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021900629999999999 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818497+0000", 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014070900000000001 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15188 3.a e41/28 3:55e8f3c2:::rbd_data.11243bbd48bb.000000000000000e:head v 41'123, mlcod=41'121)", 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:25.993184+0000", 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.695592487999999, 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002423438, 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993184+0000", 2024-09-11T05:51:37.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993184+0000", 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993186+0000", 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4930000000000001e-06 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993208+0000", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1852999999999999e-05 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993247+0000", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9403000000000002e-05 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993268+0000", 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0866999999999999e-05 2024-09-11T05:51:37.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993434+0000", 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016565500000000001 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.993495+0000", 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0749999999999999e-05 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.995495+0000", 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020005639999999998 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.995607+0000", 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111853 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.750 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15346 3.a e41/28 3:5282b638:::rbd_data.11243bbd48bb.0000000000000036:head v 41'128, mlcod=41'126)", 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.416149855, 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027676279999999998, 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272629+0000", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729999999999999e-06 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272646+0000", 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7022999999999999e-05 2024-09-11T05:51:37.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272688+0000", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2466999999999998e-05 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272709+0000", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0543e-05 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272878+0000", 2024-09-11T05:51:37.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169755 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272931+0000", 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2889999999999997e-05 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275302+0000", 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023706650000000001 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275394+0000", 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1811999999999994e-05 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15357 3.10 e41/28 3:0f623401:::rbd_data.11243bbd48bb.0000000000000009:head v 41'54, mlcod=41'52)", 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:26.280686+0000", 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.408089914, 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024559849999999999, 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280686+0000", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280686+0000", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280688+0000", 2024-09-11T05:51:37.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280709+0000", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0136000000000002e-05 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280752+0000", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3566000000000003e-05 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280772+0000", 2024-09-11T05:51:37.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0064e-05 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.280946+0000", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173788 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.281004+0000", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8035999999999998e-05 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.283013+0000", 2024-09-11T05:51:37.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020090020000000002 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.283142+0000", 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012902899999999999 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:16130 3.10 e42/28 3:094d2477:::rbd_data.1124d6faf06d.000000000000008b:head v 42'59, mlcod=42'57)", 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:27.216312+0000", 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.472463588, 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026734630000000001, 2024-09-11T05:51:37.759 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216312+0000", 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216312+0000", 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216315+0000", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1730000000000002e-06 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216331+0000", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6286e-05 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216366+0000", 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5327000000000002e-05 2024-09-11T05:51:37.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216388+0000", 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1702999999999999e-05 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216588+0000", 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019979199999999999 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216643+0000", 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5683000000000002e-05 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.218847+0000", 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022032060000000001 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.218986+0000", 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139293 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:16135 3.10 e42/28 3:0a21edc3:::rbd_data.1124d6faf06d.00000000000000d8:head v 42'60, mlcod=42'57)", 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.472239070000001, 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002949171, 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:37.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216538+0000", 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216582+0000", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3631999999999998e-05 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216607+0000", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5457e-05 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216624+0000", 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6629999999999998e-05 2024-09-11T05:51:37.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217303+0000", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000678675 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217354+0000", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1672999999999998e-05 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219332+0000", 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019773249999999998 2024-09-11T05:51:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219486+0000", 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015447200000000001 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:18462 3.1e e45/23 3:7ae7f082:::rbd_data.112453064ca6.0000000000000054:head v 45'53, mlcod=44'50)", 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.2301722430000002, 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00273369, 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458606+0000", 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2199999999999999e-06 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458699+0000", 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2737000000000003e-05 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458728+0000", 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9657000000000001e-05 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458743+0000", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4525000000000001e-05 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458902+0000", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015946600000000001 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458961+0000", 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9027000000000003e-05 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.461225+0000", 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022638469999999998 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.461337+0000", 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112211 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:51:37.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.3057980630000001, 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:51:37.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19804 3.1e e47/23 3:7c44e26f:::rbd_data.11244d01bf21.0000000000000029:head v 47'64, mlcod=47'62)", 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.7299075269999999, 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034400440000000002, 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958870+0000", 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1489999999999999e-06 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958879+0000", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5599999999999999e-06 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958898+0000", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9106999999999998e-05 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958909+0000", 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0312999999999999e-05 2024-09-11T05:51:37.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958983+0000", 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3957999999999999e-05 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.959009+0000", 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6647000000000001e-05 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:37.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962253+0000", 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032435860000000001 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962309+0000", 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5723999999999997e-05 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:37.781 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:37.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:51:37.783 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:37.783 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:37.784 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:37.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:37.785 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:37.785 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:37.781+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:37.883 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:37.881+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:37.887 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:37.887 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:37.887 INFO:teuthology.orchestra.run.smithi165.stdout:2610 read 0xf70f5a thru 0xf79fec (0x9093 bytes) 2024-09-11T05:51:37.890 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:37.890 INFO:teuthology.orchestra.run.smithi165.stdout:2613 write 0x2fecd36 thru 0x2ffc49f (0xf76a bytes) 2024-09-11T05:51:37.984 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:37.981+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:38.005 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.005 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:38.109 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:38.109+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:38.184 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.184 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.185 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:38.181+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:38.195 INFO:teuthology.orchestra.run.smithi165.stdout:2614 read 0x279a591 thru 0x27a7452 (0xcec2 bytes) 2024-09-11T05:51:38.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:38.213 INFO:teuthology.orchestra.run.smithi165.stdout:2616 write 0x9e0f8d7 thru 0x9e12227 (0x2951 bytes) 2024-09-11T05:51:38.226 INFO:teuthology.orchestra.run.smithi165.stdout:2617 write 0x954bcf8 thru 0x955ab21 (0xee2a bytes) 2024-09-11T05:51:38.258 INFO:teuthology.orchestra.run.smithi165.stdout:2618 read 0x91d2b6c thru 0x91d8865 (0x5cfa bytes) 2024-09-11T05:51:38.259 INFO:teuthology.orchestra.run.smithi165.stdout:2619 read 0x3843f8f thru 0x38511d4 (0xd246 bytes) 2024-09-11T05:51:38.263 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:38.263 INFO:teuthology.orchestra.run.smithi165.stdout:2622 read 0x104fa2d thru 0x104fc6f (0x243 bytes) 2024-09-11T05:51:38.263 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:38.263 INFO:teuthology.orchestra.run.smithi165.stdout:2629 write 0x43cf133 thru 0x43d465c (0x552a bytes) 2024-09-11T05:51:38.293 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:38.289+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:38.386 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:38.385+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:38.410 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.410 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.416 INFO:teuthology.orchestra.run.smithi165.stdout:2630 read 0x93e5813 thru 0x93f2270 (0xca5e bytes) 2024-09-11T05:51:38.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:38.429 INFO:teuthology.orchestra.run.smithi165.stdout:2631 write 0xddf7355 thru 0xddf9800 (0x24ac bytes) 2024-09-11T05:51:38.439 INFO:teuthology.orchestra.run.smithi165.stdout:2632 write 0x5332cad thru 0x5339904 (0x6c58 bytes) 2024-09-11T05:51:38.460 INFO:teuthology.orchestra.run.smithi165.stdout:2634 punch from 0x2617224 to 0x2619665, (0x2441 bytes) 2024-09-11T05:51:38.460 INFO:teuthology.orchestra.run.smithi165.stdout:2636 read 0x8f55ef5 thru 0x8f5bdc8 (0x5ed4 bytes) 2024-09-11T05:51:38.460 INFO:teuthology.orchestra.run.smithi165.stdout:2637 write 0x2ec7a1a thru 0x2ecb5cd (0x3bb4 bytes) 2024-09-11T05:51:38.464 INFO:teuthology.orchestra.run.smithi165.stdout:2638 writesame 0xc28000b thru 0xc280d4f data_size 0x1(0xd45 bytes) 2024-09-11T05:51:38.468 INFO:teuthology.orchestra.run.smithi165.stdout:2639 write 0x388ed79 thru 0x3895fee (0x7276 bytes) 2024-09-11T05:51:38.488 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:38.485+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:38.566 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.567 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:38.627 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:38.625+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:38.689 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:38.689+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:38.710 INFO:teuthology.orchestra.run.smithi165.stdout:2640 write 0x9f3f508 thru 0x9f4b3be (0xbeb7 bytes) 2024-09-11T05:51:38.727 INFO:teuthology.orchestra.run.smithi165.stdout:2641 punch from 0xa3243a0 to 0xa32e135, (0x9d95 bytes) 2024-09-11T05:51:38.727 INFO:teuthology.orchestra.run.smithi165.stdout:2642 write 0x8d0ae0b thru 0x8d113f6 (0x65ec bytes) 2024-09-11T05:51:38.748 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:38.748 INFO:teuthology.orchestra.run.smithi165.stdout:2645 read 0xc247524 thru 0xc2498ce (0x23ab bytes) 2024-09-11T05:51:38.748 INFO:teuthology.orchestra.run.smithi165.stdout:2646 writesame 0x4a9662b thru 0x4a9f687 data_size 0x3(0x905d bytes) 2024-09-11T05:51:38.749 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.749 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.772 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:51:38.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:38.791 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:38.789+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:38.815 INFO:teuthology.orchestra.run.smithi165.stdout:2648 write 0x376ab29 thru 0x3778e6b (0xe343 bytes) 2024-09-11T05:51:38.836 INFO:teuthology.orchestra.run.smithi165.stdout:2649 trunc from 0xddf9801 to 0x4822376 2024-09-11T05:51:38.881 INFO:teuthology.orchestra.run.smithi165.stdout:2650 read 0x13d34d7 thru 0x13e185a (0xe384 bytes) 2024-09-11T05:51:38.886 INFO:teuthology.orchestra.run.smithi165.stdout:2652 writesame 0xba48ab thru 0xbacf74 data_size 0x51(0x86ca bytes) 2024-09-11T05:51:38.892 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:38.889+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:38.929 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:38.929 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:38.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:38.993 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:38.993+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:39.086 INFO:teuthology.orchestra.run.smithi165.stdout:2653 trunc from 0x4822376 to 0x2a7bb33 2024-09-11T05:51:39.093 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:39.089+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:39.151 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:39.151 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:39.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:39.215 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:39.213+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:39.265 INFO:teuthology.orchestra.run.smithi165.stdout:2655 read 0x131f16f thru 0x1325c40 (0x6ad2 bytes) 2024-09-11T05:51:39.274 INFO:teuthology.orchestra.run.smithi165.stdout:2657 trunc from 0x2a7bb33 to 0x7ff2a41 2024-09-11T05:51:39.287 INFO:teuthology.orchestra.run.smithi165.stdout:2659 write 0x935ff1d thru 0x93612e9 (0x13cd bytes) 2024-09-11T05:51:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:2661 read 0x46efca4 thru 0x46f00ba (0x417 bytes) 2024-09-11T05:51:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:2662 punch from 0x2ec10e to 0x2f4f29, (0x8e1b bytes) 2024-09-11T05:51:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:2664 read 0x70a3a30 thru 0x70a883d (0x4e0e bytes) 2024-09-11T05:51:39.293 INFO:teuthology.orchestra.run.smithi165.stdout:2665 write 0x28d0a19 thru 0x28d2ca1 (0x2289 bytes) 2024-09-11T05:51:39.295 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:39.293+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:39.299 INFO:teuthology.orchestra.run.smithi165.stdout:2667 read 0x562bdf7 thru 0x562ed97 (0x2fa1 bytes) 2024-09-11T05:51:39.299 INFO:teuthology.orchestra.run.smithi165.stdout:2668 punch from 0x7edb370 to 0x7edea4a, (0x36da bytes) 2024-09-11T05:51:39.299 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:39.299 INFO:teuthology.orchestra.run.smithi165.stdout:2670 read 0x38789e2 thru 0x3879f4a (0x1569 bytes) 2024-09-11T05:51:39.309 INFO:teuthology.orchestra.run.smithi165.stdout:2671 write 0xd9df7cb thru 0xd9e3e27 (0x465d bytes) 2024-09-11T05:51:39.325 INFO:teuthology.orchestra.run.smithi165.stdout:2672 read 0x9f0a1d2 thru 0x9f12c12 (0x8a41 bytes) 2024-09-11T05:51:39.331 INFO:teuthology.orchestra.run.smithi165.stdout:2673 trunc from 0xd9e3e28 to 0x8d210b4 2024-09-11T05:51:39.340 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:39.340 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:39.349 INFO:teuthology.orchestra.run.smithi165.stdout:2675 clone 16 order 24 su 262144 sc 12 2024-09-11T05:51:39.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:39.397 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:39.397+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:39.497 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:39.493+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:39.516 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:39.516 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:39.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:39.619 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:39.617+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:39.699 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:39.693+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:39.702 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:39.702 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:39.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:39.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:39.799 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:39.797+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:39.882 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:39.883 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:39.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:39.901 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:39.901+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:39.901 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone15 from 0x8d210b4 (overlap 0x2a7bb33) to 0x10fe47a 2024-09-11T05:51:39.974 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:39.975 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:40.000 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:40.000 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:40.001 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:40.001+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:40.018 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:40.018 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:40.045 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:40.045 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:5979 3.4 e33/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 33'27, mlcod=33'25)", 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.838599631000001, 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030586089999999999, 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.051 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211937+0000", 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5499999999999998e-07 2024-09-11T05:51:40.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211943+0000", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7919999999999998e-06 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211953+0000", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.211963+0000", 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0343000000000001e-05 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212023+0000", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9224000000000001e-05 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.212049+0000", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6108000000000001e-05 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214959+0000", 2024-09-11T05:51:40.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002910564 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.214995+0000", 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6140000000000003e-05 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6057 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'16, mlcod=33'14)", 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.7356099, 2024-09-11T05:51:40.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038672799999999998, 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314927+0000", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314933+0000", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314948+0000", 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4755e-05 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.314955+0000", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5789999999999998e-06 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315012+0000", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.681e-05 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.315033+0000", 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318754+0000", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720409 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.318794+0000", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0361000000000002e-05 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6102 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'19, mlcod=33'17)", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:40.059 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.641067864, 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031722349999999998, 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409469+0000", 2024-09-11T05:51:40.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409470+0000", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1000000000000001e-06 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409480+0000", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049e-05 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409502+0000", 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1336e-05 2024-09-11T05:51:40.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409516+0000", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4793e-05 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409686+0000", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169729 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.409740+0000", 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-05 2024-09-11T05:51:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412595+0000", 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028555989999999999 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.412641+0000", 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5448000000000003e-05 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6259 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'33, mlcod=33'31)", 2024-09-11T05:51:40.063 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.36932058, 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028678200000000001, 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681216+0000", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681218+0000", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681228+0000", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0067e-05 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681246+0000", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8561999999999999e-05 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681255+0000", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681351+0000", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5390000000000007e-05 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.681365+0000", 2024-09-11T05:51:40.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4684999999999999e-05 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.683978+0000", 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026128610000000002 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.684084+0000", 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105537 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.067 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6345 3.9 e33/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 33'24, mlcod=33'22)", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.245301606000002, 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028155179999999999, 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805235+0000", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805237+0000", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.854e-06 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805249+0000", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208999999999999e-05 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805264+0000", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4738e-05 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805272+0000", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7749999999999998e-06 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805330+0000", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7327999999999997e-05 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.805347+0000", 2024-09-11T05:51:40.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7010000000000001e-05 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808014+0000", 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002666716 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.808050+0000", 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6888000000000003e-05 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6403 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'34, mlcod=33'32)", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.17030943, 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037706139999999998, 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880227+0000", 2024-09-11T05:51:40.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880229+0000", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5230000000000001e-06 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880260+0000", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1426e-05 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880286+0000", 2024-09-11T05:51:40.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5800999999999999e-05 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880296+0000", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0482000000000001e-05 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880395+0000", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8459e-05 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.880416+0000", 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1401000000000001e-05 2024-09-11T05:51:40.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883950+0000", 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035335459999999998 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:16.883998+0000", 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7976000000000003e-05 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:6714 3.4 e33/23 3:202bf89b:::rbd_data.11249b359231.0000000000000011:head v 33'41, mlcod=33'39)", 2024-09-11T05:51:40.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.696334310000001, 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003475395, 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354202+0000", 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354205+0000", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354218+0000", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3234000000000001e-05 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354253+0000", 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4830999999999999e-05 2024-09-11T05:51:40.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354269+0000", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6677999999999999e-05 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354397+0000", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012779499999999999 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.354451+0000", 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4014000000000002e-05 2024-09-11T05:51:40.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357590+0000", 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031385060000000001 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.357678+0000", 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7973000000000006e-05 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7021 3.9 e34/23 3:917dc6f1:::rbd_data.11249b359231.0000000000000024:head v 34'34, mlcod=34'32)", 2024-09-11T05:51:40.079 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.203705046, 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034247660000000001, 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846831+0000", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846832+0000", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6400000000000001e-07 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.846839+0000", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8190000000000003e-06 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847228+0000", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038838799999999998 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847237+0000", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2289999999999992e-06 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847302+0000", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5228999999999995e-05 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.847328+0000", 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5675000000000001e-05 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850198+0000", 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028700309999999999 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:17.850256+0000", 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8530999999999999e-05 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7116 3.9 e34/23 3:938b1af5:::rbd_data.11249b359231.0000000000000001:head v 34'35, mlcod=34'33)", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.043477069000001, 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00371405, 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007059+0000", 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007061+0000", 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5090000000000001e-06 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007072+0000", 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1399e-05 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007100+0000", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7531000000000001e-05 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007114+0000", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4378e-05 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007340+0000", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022568499999999999 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.007387+0000", 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7194000000000002e-05 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010717+0000", 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033296369999999999 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.010773+0000", 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6716999999999999e-05 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:7119 3.4 e34/23 3:25de1119:::rbd_data.11249b359231.0000000000000004:head v 34'49, mlcod=34'47)", 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.042271290999999, 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034162110000000002, 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008265+0000", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008267+0000", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.294e-06 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008276+0000", 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5820000000000008e-06 2024-09-11T05:51:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008290+0000", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4005000000000001e-05 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008298+0000", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3189999999999995e-06 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008368+0000", 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0055999999999995e-05 2024-09-11T05:51:40.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.008396+0000", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7563999999999999e-05 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011628+0000", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032314969999999998 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:18.011681+0000", 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3894000000000003e-05 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.091 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:12734 3.0 e38/23 3:06e59eda:::rbd_data.1124518b2c12.000000000000002b:head v 38'41, mlcod=37'39)", 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.240261287999999, 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033376479999999999, 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810275+0000", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810277+0000", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4699999999999999e-06 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810314+0000", 2024-09-11T05:51:40.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7370999999999998e-05 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810336+0000", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2129000000000001e-05 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810347+0000", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0465999999999999e-05 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810418+0000", 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099000000000005e-05 2024-09-11T05:51:40.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.810443+0000", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4825999999999999e-05 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813553+0000", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031099790000000001 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:22.813613+0000", 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0307999999999999e-05 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15206 3.0 e41/23 3:01a205d5:::rbd_data.11243bbd48bb.0000000000000004:head v 41'65, mlcod=41'63)", 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.046561057, 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034631129999999999, 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003975+0000", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003977+0000", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.003986+0000", 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6120000000000005e-06 2024-09-11T05:51:40.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004008+0000", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1679e-05 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004018+0000", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5210000000000008e-06 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004105+0000", 2024-09-11T05:51:40.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7371000000000001e-05 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.004128+0000", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2682999999999999e-05 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007395+0000", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032676129999999999 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.007439+0000", 2024-09-11T05:51:40.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3306999999999997e-05 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15537 3.4 e42/23 3:242e3831:::rbd_data.11243bbd48bb.000000000000001e:head v 42'64, mlcod=41'61)", 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.287441707999999, 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031185620000000001, 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763095+0000", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763096+0000", 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763105+0000", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2119999999999992e-06 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763125+0000", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9536999999999999e-05 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763135+0000", 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0022e-05 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763216+0000", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1606000000000004e-05 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.763236+0000", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9585e-05 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766184+0000", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002948169 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.766213+0000", 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.917e-05 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:15715 3.0 e42/23 3:0151ec39:::rbd_data.1124d6faf06d.000000000000006a:head v 42'71, mlcod=42'67)", 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.130733114, 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028945469999999999, 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919803+0000", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919804+0000", 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-07 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919810+0000", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8919999999999997e-06 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919823+0000", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2879e-05 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919830+0000", 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.635e-06 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919877+0000", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7049999999999998e-05 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.919896+0000", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9216999999999999e-05 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922658+0000", 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002762675 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:26.922698+0000", 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9477000000000002e-05 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:16482 3.9 e43/23 3:975fce18:::rbd_data.1124d6faf06d.0000000000000000:head v 43'50, mlcod=40'48)", 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 12.246996376, 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046884810000000004, 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.108 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803540+0000", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.803541+0000", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8899999999999999e-07 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.804985+0000", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014445459999999999 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805024+0000", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8390999999999999e-05 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805036+0000", 2024-09-11T05:51:40.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1799e-05 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805107+0000", 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1530000000000001e-05 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.805135+0000", 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8384000000000001e-05 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808057+0000", 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029216979999999999 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:27.808229+0000", 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017144400000000001 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(3.9 epoch 47/23 rep_tid 602 entries 47'74 (0'0) error 3:957a1c2c:::rbd_data.11244d01bf21.0000000000000016:head by client.4388.0:19596 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 46'72)", 2024-09-11T05:51:40.112 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.114468767, 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030496590000000001, 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936068+0000", 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.31e-07 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936074+0000", 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1870000000000002e-06 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936127+0000", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3325999999999997e-05 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936151+0000", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3354e-05 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.936266+0000", 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115061 2024-09-11T05:51:40.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.939117+0000", 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028516000000000001 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:19761 3.9 e47/23 3:97eabedf:::rbd_data.11244d01bf21.0000000000000021:head v 47'76, mlcod=46'73)", 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.1065129599999999, 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051400339999999999, 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.116 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944024+0000", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944025+0000", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944036+0000", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0757999999999999e-05 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944052+0000", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6565000000000001e-05 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944061+0000", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7479999999999999e-06 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944130+0000", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9088e-05 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.944154+0000", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4541000000000001e-05 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.949044+0000", 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048898650000000002 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.949164+0000", 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119232 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:19824 3.9 e47/23 3:971d1760:::rbd_data.11244d01bf21.000000000000003c:head v 47'81, mlcod=47'77)", 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.0792666430000004, 2024-09-11T05:51:40.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035938300000000001, 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971270+0000", 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971271+0000", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.102e-06 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971279+0000", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8790000000000003e-06 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971298+0000", 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8771e-05 2024-09-11T05:51:40.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971307+0000", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2010000000000005e-06 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971394+0000", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6780000000000006e-05 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.971424+0000", 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.082e-05 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.974820+0000", 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003395726 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:31.974864+0000", 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3550999999999997e-05 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.124 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:27938 3.4 e48/23 3:230d7789:::rbd_data.1124a9e466a.0000000000000000:head v 48'111, mlcod=48'109)", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:35.226778+0000", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8237585359999997, 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034323550000000002, 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226778+0000", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226778+0000", 2024-09-11T05:51:40.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226779+0000", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3650000000000001e-06 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226792+0000", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2490000000000001e-05 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226816+0000", 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4138000000000002e-05 2024-09-11T05:51:40.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226826+0000", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0502e-05 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226923+0000", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7015999999999994e-05 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.226948+0000", 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4394000000000001e-05 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.230157+0000", 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032094480000000002 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:35.230210+0000", 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3001999999999998e-05 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.030129827000000001, 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:51:40.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:51:40.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:51:40.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.132 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:40.133 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:40.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:40.134 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:40.135 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:40.135 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:40.135 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:40.135 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:40.135 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:40.129+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:40.151 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:40.151 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:40.240 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:40.233+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:40.320 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:40.320 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:40.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64047 3.2 3:450f9540:::rbd_data.1124a9e466a.0000000000000006:10 [read 1753088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.317906+0000", 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0022179600000000002, 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022461489999999998, 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64047 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.321 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.317906+0000", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.317906+0000", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.317906+0000", 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-11T05:51:40.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.317909+0000", 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8150000000000002e-06 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320020+0000", 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021110909999999998 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320027+0000", 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.81e-06 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320080+0000", 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2954999999999997e-05 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320123+0000", 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2861999999999997e-05 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320163+0000", 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9616999999999997e-05 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64050 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320049+0000", 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.5183000000000002e-05, 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019124499999999999, 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.325 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64050 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320049+0000", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320049+0000", 2024-09-11T05:51:40.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320049+0000", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3499999999999995e-07 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320053+0000", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8979999999999998e-06 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320063+0000", 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2869999999999993e-06 2024-09-11T05:51:40.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320069+0000", 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0839999999999997e-06 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64056 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320111+0000", 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.2625e-05, 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018446599999999999, 2024-09-11T05:51:40.328 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64056 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320111+0000", 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320111+0000", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320112+0000", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7899999999999998e-07 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320115+0000", 2024-09-11T05:51:40.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320124+0000", 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8219999999999996e-06 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320130+0000", 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1619999999999996e-06 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64062 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320141+0000", 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999833, 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021320900000000001, 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64062 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320141+0000", 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320141+0000", 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320141+0000", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7199999999999999e-07 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320143+0000", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4040000000000002e-06 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320150+0000", 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.866e-06 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320156+0000", 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4450000000000004e-06 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64063 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320164+0000", 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999599, 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027168800000000002, 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.335 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64063 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320164+0000", 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320164+0000", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320164+0000", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320167+0000", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6390000000000001e-06 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320176+0000", 2024-09-11T05:51:40.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8009999999999993e-06 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320182+0000", 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5230000000000001e-06 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64069 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320191+0000", 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9999328, 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310437, 2024-09-11T05:51:40.338 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64069 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320191+0000", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320191+0000", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320192+0000", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.39e-07 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320194+0000", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6120000000000001e-06 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320202+0000", 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1049999999999995e-06 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320208+0000", 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1469999999999998e-06 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64075 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.341 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320217+0000", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.999907, 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034126600000000001, 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64075 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320217+0000", 2024-09-11T05:51:40.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320217+0000", 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320217+0000", 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3099999999999998e-07 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320220+0000", 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3149999999999999e-06 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320228+0000", 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9589999999999995e-06 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320233+0000", 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2830000000000001e-06 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64084 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320240+0000", 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998846, 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039533199999999998, 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64084 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320240+0000", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320240+0000", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320240+0000", 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0999999999999999e-07 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320242+0000", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.238e-06 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320250+0000", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7959999999999996e-06 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320255+0000", 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3580000000000002e-06 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64090 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320262+0000", 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998617, 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000413936, 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64090 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.348 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320262+0000", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320262+0000", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320263+0000", 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5600000000000001e-07 2024-09-11T05:51:40.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320265+0000", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2019999999999998e-06 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320277+0000", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1584e-05 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320282+0000", 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6409999999999999e-06 2024-09-11T05:51:40.350 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64097 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320289+0000", 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998345, 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043729400000000002, 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.351 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64097 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320289+0000", 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320289+0000", 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320290+0000", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9200000000000001e-07 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320292+0000", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3140000000000002e-06 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320300+0000", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7959999999999996e-06 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320305+0000", 2024-09-11T05:51:40.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4310000000000002e-06 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64103 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320312+0000", 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998116, 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049185400000000001, 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64103 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320312+0000", 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320312+0000", 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320313+0000", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3000000000000001e-07 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320315+0000", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2359999999999999e-06 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320322+0000", 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5089999999999996e-06 2024-09-11T05:51:40.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320328+0000", 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1709999999999999e-06 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64110 3.8 3.7fe792c8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320337+0000", 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9997869, 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055065599999999995, 2024-09-11T05:51:40.357 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64110 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320337+0000", 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320337+0000", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320338+0000", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320340+0000", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2110000000000001e-06 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320348+0000", 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0029999999999994e-06 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320354+0000", 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0859999999999998e-06 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64117 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320362+0000", 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9997621, 2024-09-11T05:51:40.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000586617, 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64117 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320362+0000", 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320362+0000", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320362+0000", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320364+0000", 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1430000000000001e-06 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320372+0000", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7470000000000005e-06 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320378+0000", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5010000000000004e-06 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64118 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320384+0000", 2024-09-11T05:51:40.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9997396, 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000627237, 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64118 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320384+0000", 2024-09-11T05:51:40.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320384+0000", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320385+0000", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6600000000000002e-07 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320394+0000", 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9439999999999994e-06 2024-09-11T05:51:40.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320402+0000", 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4079999999999998e-06 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320408+0000", 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3789999999999997e-06 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.366 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64124 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320463+0000", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9996605, 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060731299999999995, 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64124 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320463+0000", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320463+0000", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320464+0000", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9199999999999999e-07 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320469+0000", 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8790000000000002e-06 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320479+0000", 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320485+0000", 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.545e-06 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64130 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320521+0000", 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9996023, 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060491799999999997, 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64130 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320521+0000", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320521+0000", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320522+0000", 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9999999999999998e-07 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320524+0000", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3479999999999998e-06 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320531+0000", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5139999999999998e-06 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320536+0000", 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64140 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320564+0000", 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9995604, 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063161900000000004, 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64140 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320564+0000", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320564+0000", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320564+0000", 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320566+0000", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1849999999999998e-06 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320573+0000", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1210000000000001e-06 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320580+0000", 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0820000000000004e-06 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.375 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64147 3.18 3.1d709bb8 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320606+0000", 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9995179, 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065177300000000002, 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64147 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320606+0000", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320606+0000", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320606+0000", 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0699999999999998e-07 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320608+0000", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0600000000000002e-06 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320615+0000", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5200000000000003e-06 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320620+0000", 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9099999999999996e-06 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:64154 3.2 3:450f9540:::rbd_data.1124a9e466a.0000000000000006:10 [read 1349632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.320682+0000", 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9994421, 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064128099999999995, 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 64154 2024-09-11T05:51:40.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320682+0000", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320682+0000", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320683+0000", 2024-09-11T05:51:40.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320686+0000", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1080000000000001e-06 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320694+0000", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7249999999999997e-06 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.320699+0000", 2024-09-11T05:51:40.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7509999999999997e-06 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.321173+0000", 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047378299999999997 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.321239+0000", 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6204000000000005e-05 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.382 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:40.383 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 19 2024-09-11T05:51:40.383 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:40.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:40.393 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:40.393 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:40.393 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.29086991077938645 2024-09-11T05:51:40.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.29086991077938645 2024-09-11T05:51:40.394 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:40.381+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:40.484 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:40.481+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:40.514 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:40.514 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:40.514 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:40.514 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:40.514 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:40.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:40.559 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-09-11T05:51:40.603 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:40.601+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:40.687 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:40.681+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20508 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 131072~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3789374549999991, 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020384915, 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.695 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20508 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314480+0000", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314481+0000", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.083e-06 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314487+0000", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8030000000000002e-06 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314505+0000", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7779e-05 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314518+0000", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3475999999999999e-05 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334353+0000", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019834766 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334592+0000", 2024-09-11T05:51:40.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023897499999999999 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334865+0000", 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000273033 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20512 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3788812250000007, 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020911447, 2024-09-11T05:51:40.699 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20512 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314536+0000", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314537+0000", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.023e-06 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314544+0000", 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2060000000000004e-06 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314562+0000", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7992e-05 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.314574+0000", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2716e-05 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.334986+0000", 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020411838000000002 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335179+0000", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019315000000000001 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.335448+0000", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026852199999999999 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20522 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 524288~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3775153309999997, 2024-09-11T05:51:40.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020763599000000001, 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20522 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315902+0000", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315904+0000", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.33e-06 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315911+0000", 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7530000000000001e-06 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315933+0000", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1868000000000001e-05 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315949+0000", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6146000000000001e-05 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336229+0000", 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020279993999999999 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336424+0000", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019419000000000001 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336666+0000", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024231799999999999 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20524 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:40.707 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3774453290000004, 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021361196999999998, 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20524 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315972+0000", 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315974+0000", 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.367e-06 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.315981+0000", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3690000000000001e-06 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316034+0000", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2694000000000001e-05 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316053+0000", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9324999999999999e-05 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.336785+0000", 2024-09-11T05:51:40.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020731611 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337045+0000", 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000260287 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337333+0000", 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028854400000000001 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20528 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 802816~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3772822120000008, 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021755004000000001, 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20528 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316135+0000", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316137+0000", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6929999999999999e-06 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316144+0000", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3889999999999999e-06 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316166+0000", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1685999999999999e-05 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316183+0000", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6509000000000001e-05 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337439+0000", 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021256744000000001 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337660+0000", 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022058000000000001 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337890+0000", 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023040300000000001 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20529 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 917504~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3772114700000007, 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022138649999999999, 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20529 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316206+0000", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316208+0000", 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4899999999999999e-06 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316220+0000", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2629000000000001e-05 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316240+0000", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9664999999999999e-05 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316255+0000", 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929000000000001e-05 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.337958+0000", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021703009999999998 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338131+0000", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000172991 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338345+0000", 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021393599999999999 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:20532 3.b 3:d43a484b:::rbd_data.1124d6faf06d.0000000000000002:c [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3771428080000003, 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022552875, 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 20532 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.720 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316275+0000", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316276+0000", 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.04e-06 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316282+0000", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7869999999999998e-06 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316300+0000", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7965000000000002e-05 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.316314+0000", 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4382999999999999e-05 2024-09-11T05:51:40.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338416+0000", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022102087999999999 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338602+0000", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000186212 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.338828+0000", 2024-09-11T05:51:40.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0002254 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23877 3.7 3:e4bc48c1:::rbd_data.11244d01bf21.000000000000002f:f [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.0069168990000001, 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022057406000000002, 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23877 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686501+0000", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4400000000000001e-07 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686503+0000", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1490000000000001e-06 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686510+0000", 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3419999999999996e-06 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686514+0000", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7350000000000004e-06 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.702863+0000", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016348260999999999 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.702934+0000", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1564999999999997e-05 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.708558+0000", 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056239100000000002 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23912 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.0066288130000007, 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020603792999999999, 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23912 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686789+0000", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3000000000000002e-07 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686791+0000", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.863e-06 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686796+0000", 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4249999999999997e-06 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686801+0000", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2379999999999999e-06 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.706863+0000", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020062582999999998 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.706936+0000", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3106000000000001e-05 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707393+0000", 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045624800000000003 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:23913 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 917504~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.0066104869999997, 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020767105000000001, 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 23913 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686807+0000", 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686808+0000", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8800000000000002e-07 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686810+0000", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6450000000000001e-06 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686817+0000", 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.686822+0000", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6739999999999996e-06 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707416+0000", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020593923 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707492+0000", 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6050000000000005e-05 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.707574+0000", 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2163999999999995e-05 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24364 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 278528~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9953814799999998, 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021347893999999999, 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24364 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698036+0000", 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698037+0000", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8100000000000003e-07 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698039+0000", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2680000000000001e-06 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698045+0000", 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1419999999999998e-06 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698050+0000", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7469999999999996e-06 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.714965+0000", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016914769 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.717633+0000", 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026687899999999999 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719384+0000", 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001750597 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24365 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 352256~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9953615510000002, 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021530918999999999, 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.740 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24365 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698056+0000", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6199999999999999e-07 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698058+0000", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0320000000000002e-06 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698064+0000", 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0090000000000004e-06 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698069+0000", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7389999999999999e-06 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719418+0000", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021349283 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719501+0000", 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2282999999999998e-05 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719587+0000", 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6211000000000005e-05 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24366 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9953421770000004, 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021705247, 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24366 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698075+0000", 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698076+0000", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8299999999999997e-07 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698078+0000", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0379999999999998e-06 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698084+0000", 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.913e-06 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698089+0000", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6820000000000002e-06 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719617+0000", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021528623 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719696+0000", 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8755000000000001e-05 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719781+0000", 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4653000000000004e-05 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24367 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9953230269999995, 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021876105999999999, 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.748 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24367 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698095+0000", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698097+0000", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9080000000000002e-06 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698103+0000", 2024-09-11T05:51:40.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.879e-06 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698107+0000", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7199999999999997e-06 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719811+0000", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021703940000000001 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719887+0000", 2024-09-11T05:51:40.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6081000000000006e-05 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.719971+0000", 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3208999999999996e-05 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24368 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 704512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9953040909999995, 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022050626, 2024-09-11T05:51:40.752 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24368 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698114+0000", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7300000000000001e-07 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698116+0000", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3910000000000001e-06 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698122+0000", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0680000000000002e-06 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698127+0000", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720001+0000", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021873373000000002 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720078+0000", 2024-09-11T05:51:40.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7645000000000006e-05 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720164+0000", 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5667999999999994e-05 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24369 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 778240~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9952825770000002, 2024-09-11T05:51:40.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023716779, 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24369 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698135+0000", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698136+0000", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698138+0000", 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3980000000000002e-06 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698145+0000", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8220000000000001e-06 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698150+0000", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9830000000000004e-06 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720194+0000", 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022044587000000001 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.720270+0000", 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6077999999999999e-05 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721852+0000", 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001581442 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24371 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 851968~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.760 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9952393979999998, 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023858969000000001, 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24371 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698178+0000", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698179+0000", 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698181+0000", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9910000000000001e-06 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698186+0000", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7520000000000002e-06 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698191+0000", 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2939999999999999e-06 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721888+0000", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023697685999999999 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.721965+0000", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7156000000000005e-05 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722037+0000", 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1717999999999993e-05 2024-09-11T05:51:40.764 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:24372 3.7 3:e32e6f3a:::rbd_data.11244d01bf21.0000000000000032:f [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.995220818, 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023933343999999999, 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.765 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 24372 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698197+0000", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5999999999999999e-07 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698203+0000", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2920000000000001e-06 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698210+0000", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6800000000000004e-06 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.698215+0000", 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6519999999999997e-06 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722055+0000", 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023840296 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722096+0000", 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0775000000000002e-05 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:32.722130+0000", 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4289000000000003e-05 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:29200 3.8 3:1216e164:::rbd_data.1124b2f65e38.000000000000008e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:36.210516+0000", 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.4829016529999999, 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023560235999999998, 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.769 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 29200 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210516+0000", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210516+0000", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210517+0000", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210525+0000", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3379999999999999e-06 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210542+0000", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7918999999999999e-05 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210552+0000", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8859999999999999e-06 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210614+0000", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1156999999999997e-05 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.210724+0000", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110728 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.214127+0000", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034030319999999998 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.233875+0000", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019748109999999999 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.233907+0000", 2024-09-11T05:51:40.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1334999999999997e-05 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.234005+0000", 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8574999999999997e-05 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.234076+0000", 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0876000000000004e-05 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:30287 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 4456448~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:36.889136+0000", 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8042817429999998, 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021701133000000001, 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 30287 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889136+0000", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889136+0000", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889137+0000", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4330000000000001e-06 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:40.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889144+0000", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2269999999999998e-06 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889162+0000", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8212000000000001e-05 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.889175+0000", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3506000000000001e-05 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.910387+0000", 2024-09-11T05:51:40.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021212242999999999 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.910563+0000", 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000175487 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:36.910837+0000", 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027402500000000001 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:40.778 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:40.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:40.780 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:40.780 INFO:teuthology.orchestra.run.smithi165.stdout:2679 read 0x8824928 thru 0x88292c7 (0x49a0 bytes) 2024-09-11T05:51:40.780 INFO:teuthology.orchestra.run.smithi165.stdout:2680 writesame 0xdd4ad6f thru 0xdd4b1fc data_size 0xb(0x48e bytes) 2024-09-11T05:51:40.780 INFO:teuthology.orchestra.run.smithi165.stdout:2681 read 0xfc9faa thru 0xfd7e9f (0xdef6 bytes) 2024-09-11T05:51:40.781 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:40.781 INFO:teuthology.orchestra.run.smithi165.stdout:2683 write 0xa51b81b thru 0xa52a1e5 (0xe9cb bytes) 2024-09-11T05:51:40.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:51:40.824 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:40.821+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:40.877 INFO:teuthology.orchestra.run.smithi165.stdout:2684 write 0x9ed4b67 thru 0x9ee2e88 (0xe322 bytes) 2024-09-11T05:51:41.021 INFO:teuthology.orchestra.run.smithi165.stdout:2686 trunc from 0xdd4b1fd to 0x9447006 2024-09-11T05:51:41.022 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:70318 3.14 3:28a25632:::rbd_object_map.1124c865af2a:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.927888+0000", 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.000886643, 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092572900000000003, 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 70318 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927888+0000", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927888+0000", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927889+0000", 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7710000000000001e-06 2024-09-11T05:51:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927898+0000", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9549999999999998e-06 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927913+0000", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4657999999999999e-05 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927920+0000", 2024-09-11T05:51:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3209999999999999e-06 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.927995+0000", 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4414999999999998e-05 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.928084+0000", 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9012999999999996e-05 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.928402+0000", 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031819200000000002 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-11T05:51:41.027 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:41.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:41.028 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:41.017+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:41.123 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:41.117+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:41.201 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.201 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:41.202 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:41.202 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:41.202 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:41.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:41.224 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:41.221+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:41.325 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:41.321+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:41.337 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58950 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2557345770000001, 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064766338000000007, 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.338 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58950 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079620+0000", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1880000000000001e-06 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079627+0000", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2740000000000004e-06 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079633+0000", 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1770000000000003e-06 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144235+0000", 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064602407000000001 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144301+0000", 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6125999999999997e-05 2024-09-11T05:51:41.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144384+0000", 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2696999999999998e-05 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58951 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 966656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.255712712, 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064870423999999996, 2024-09-11T05:51:41.342 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58951 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079642+0000", 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2910000000000002e-06 2024-09-11T05:51:41.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079649+0000", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999999e-06 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079653+0000", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9439999999999996e-06 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144413+0000", 2024-09-11T05:51:41.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064759540000000004 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144455+0000", 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2438000000000003e-05 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144510+0000", 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4573000000000001e-05 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58952 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2556832010000001, 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065131952000000007, 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58952 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079670+0000", 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079672+0000", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.627e-06 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079680+0000", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6590000000000007e-06 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079685+0000", 2024-09-11T05:51:41.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3469999999999998e-06 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144541+0000", 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064855876000000007 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144725+0000", 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018391499999999999 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144801+0000", 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5826000000000002e-05 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58953 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.255659308, 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065264442000000006, 2024-09-11T05:51:41.351 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58953 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079694+0000", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2799999999999995e-07 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079696+0000", 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0789999999999999e-06 2024-09-11T05:51:41.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079702+0000", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.455e-06 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079708+0000", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7749999999999998e-06 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144832+0000", 2024-09-11T05:51:41.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065124390000000004 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144887+0000", 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4669000000000001e-05 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144957+0000", 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0345999999999994e-05 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58954 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.255637669, 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066242835, 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58954 2024-09-11T05:51:41.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:41.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.08e-07 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079723+0000", 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2830000000000001e-06 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079730+0000", 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6059999999999999e-06 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079735+0000", 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1930000000000001e-06 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145779+0000", 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066043973000000006 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145861+0000", 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1984000000000007e-05 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145957+0000", 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6387999999999999e-05 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58959 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2015232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2556091140000001, 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065383153999999999, 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.360 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58959 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079744+0000", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079746+0000", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4830000000000002e-06 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079759+0000", 2024-09-11T05:51:41.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3113e-05 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079766+0000", 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.832e-06 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144985+0000", 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065218999999999999 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145060+0000", 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5186999999999997e-05 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145126+0000", 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6126999999999999e-05 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.364 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58966 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2093056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.255578477, 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065511660999999999, 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58966 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999996e-07 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079777+0000", 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6740000000000001e-06 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079782+0000", 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7559999999999999e-06 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079787+0000", 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3580000000000002e-06 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145153+0000", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065365728999999997 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145219+0000", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5720000000000001e-05 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145285+0000", 2024-09-11T05:51:41.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6927000000000005e-05 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58972 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2109440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2555603200000001, 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065642100999999994, 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.369 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58972 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080613+0000", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00082077599999999995 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080617+0000", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080627+0000", 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.004e-05 2024-09-11T05:51:41.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080634+0000", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7519999999999999e-06 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145311+0000", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064677068000000004 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145367+0000", 2024-09-11T05:51:41.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6137999999999997e-05 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145434+0000", 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6931e-05 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58978 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2187264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.254695779, 2024-09-11T05:51:41.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064940567000000005, 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58978 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:41.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080660+0000", 2024-09-11T05:51:41.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8629999999999999e-06 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080668+0000", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0879999999999995e-06 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080682+0000", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3788000000000001e-05 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145461+0000", 2024-09-11T05:51:41.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064779310000000007 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145528+0000", 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6764e-05 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145597+0000", 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9052000000000002e-05 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58985 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2234368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.254660377, 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065076868999999996, 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58985 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080695+0000", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4959999999999999e-06 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080702+0000", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3529999999999997e-06 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080708+0000", 2024-09-11T05:51:41.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145624+0000", 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064915918000000003 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145693+0000", 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8759999999999999e-05 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145769+0000", 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5862e-05 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58991 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2312192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.254636842, 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065155451000000003, 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.382 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58991 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:41.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080716+0000", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2500000000000001e-07 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080718+0000", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080725+0000", 2024-09-11T05:51:41.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080730+0000", 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2989999999999996e-06 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145793+0000", 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065062732999999998 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145831+0000", 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8136999999999998e-05 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145871+0000", 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9745e-05 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58997 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2546153689999999, 2024-09-11T05:51:41.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065296924000000006, 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58997 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080837+0000", 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9950000000000004e-05 2024-09-11T05:51:41.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080841+0000", 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2610000000000004e-06 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080850+0000", 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5639999999999997e-06 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080856+0000", 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.669e-06 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145895+0000", 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065038809000000003 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145956+0000", 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0461999999999998e-05 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146034+0000", 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8208999999999996e-05 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58998 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2390016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2544849849999999, 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065346670999999995, 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.391 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58998 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081070+0000", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020277699999999999 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081075+0000", 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.972e-06 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081085+0000", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9459999999999993e-06 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081093+0000", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9060000000000002e-06 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146084+0000", 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064990669000000001 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146155+0000", 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1252000000000002e-05 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146214+0000", 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9148999999999999e-05 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.395 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59006 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2140160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.254244983, 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065251679000000007, 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59006 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.396 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081305+0000", 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197519 2024-09-11T05:51:41.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081309+0000", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081318+0000", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7900000000000005e-06 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081325+0000", 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4950000000000002e-06 2024-09-11T05:51:41.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146244+0000", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064918645999999997 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146303+0000", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8668000000000001e-05 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146359+0000", 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6471999999999997e-05 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.399 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59012 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2217984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.254008891, 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065189569000000003, 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59012 2024-09-11T05:51:41.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082116+0000", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077256999999999998 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082144+0000", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7863000000000001e-05 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082163+0000", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9378999999999999e-05 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082195+0000", 2024-09-11T05:51:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1782999999999997e-05 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146393+0000", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064197562999999999 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146460+0000", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7197000000000001e-05 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146533+0000", 2024-09-11T05:51:41.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3213999999999994e-05 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59024 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2342912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2531165, 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064656276999999998, 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59024 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082244+0000", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1999999999999994e-06 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082252+0000", 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.435e-06 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082258+0000", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8610000000000003e-06 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146738+0000", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064479936000000002 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146814+0000", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6728000000000001e-05 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146892+0000", 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7645999999999995e-05 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59031 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2420736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2530864859999999, 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064743414999999999, 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59031 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.63e-07 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082269+0000", 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2670000000000001e-06 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082275+0000", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2759999999999997e-06 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082280+0000", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9269999999999996e-06 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146928+0000", 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064647901999999993 2024-09-11T05:51:41.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146975+0000", 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6860000000000002e-05 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147009+0000", 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.472e-05 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59039 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2437120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.253065946, 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848283000000007, 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59039 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082342+0000", 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5473000000000002e-05 2024-09-11T05:51:41.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082346+0000", 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8310000000000003e-06 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082354+0000", 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6549999999999993e-06 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082361+0000", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147025+0000", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064663913000000003 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147082+0000", 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6805000000000002e-05 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147135+0000", 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2438999999999997e-05 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59045 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2514944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.252980569, 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064917090999999996, 2024-09-11T05:51:41.417 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59045 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083064+0000", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069213100000000002 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083069+0000", 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2789999999999998e-06 2024-09-11T05:51:41.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083081+0000", 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2251e-05 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083089+0000", 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8140000000000001e-06 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147163+0000", 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064073571999999995 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147228+0000", 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5235999999999997e-05 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147289+0000", 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0807999999999998e-05 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59068 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2592768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.2519868189999999, 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064977262999999993, 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59068 2024-09-11T05:51:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.084910+0000", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001544836 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085422+0000", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051133000000000005 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085433+0000", 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-11T05:51:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085441+0000", 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179999999999998e-06 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148208+0000", 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062766695999999997 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148273+0000", 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5395999999999995e-05 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148343+0000", 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9807000000000004e-05 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.426 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:41.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:41.428 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:41.425+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:41.528 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:41.524+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:41.533 INFO:teuthology.orchestra.run.smithi165.stdout:2687 read 0x5ef1826 thru 0x5efb5e3 (0x9dbe bytes) 2024-09-11T05:51:41.534 INFO:teuthology.orchestra.run.smithi165.stdout:2688 read 0x8196d20 thru 0x819dcd3 (0x6fb4 bytes) 2024-09-11T05:51:41.534 INFO:teuthology.orchestra.run.smithi165.stdout:2689 punch from 0x70e1d02 to 0x70e78a9, (0x5ba7 bytes) 2024-09-11T05:51:41.534 INFO:teuthology.orchestra.run.smithi165.stdout:2692 write 0x6500072 thru 0x6507001 (0x6f90 bytes) 2024-09-11T05:51:41.555 INFO:teuthology.orchestra.run.smithi165.stdout:2694 write 0xdbda8bd thru 0xdbdc633 (0x1d77 bytes) 2024-09-11T05:51:41.558 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.558 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:41.558 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:41.558 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:41.561 INFO:teuthology.orchestra.run.smithi165.stdout:2695 write 0x11ee0f9 thru 0x11fd04b (0xef53 bytes) 2024-09-11T05:51:41.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:41.575 INFO:teuthology.orchestra.run.smithi165.stdout:2697 write 0xe9d3b5d thru 0xe9d7bf5 (0x4099 bytes) 2024-09-11T05:51:41.580 INFO:teuthology.orchestra.run.smithi165.stdout:2698 read 0x48c3f2d thru 0x48c498f (0xa63 bytes) 2024-09-11T05:51:41.580 INFO:teuthology.orchestra.run.smithi165.stdout:2699 punch from 0x54b462a to 0x54bed7e, (0xa754 bytes) 2024-09-11T05:51:41.580 INFO:teuthology.orchestra.run.smithi165.stdout:2700 writesame 0x21dc78c thru 0x21de3dd data_size 0x1d(0x1c52 bytes) 2024-09-11T05:51:41.583 INFO:teuthology.orchestra.run.smithi165.stdout:2703 write 0x6857da3 thru 0x685b63a (0x3898 bytes) 2024-09-11T05:51:41.589 INFO:teuthology.orchestra.run.smithi165.stdout:2705 trunc from 0xe9d7bf6 to 0xdc1b635 2024-09-11T05:51:41.601 INFO:teuthology.orchestra.run.smithi165.stdout:2706 punch from 0x8e2e419 to 0x8e3d446, (0xf02d bytes) 2024-09-11T05:51:41.601 INFO:teuthology.orchestra.run.smithi165.stdout:2707 punch from 0x4dd412e to 0x4ddc3d9, (0x82ab bytes) 2024-09-11T05:51:41.601 INFO:teuthology.orchestra.run.smithi165.stdout:2708 write 0x2d8df6b thru 0x2d90fa6 (0x303c bytes) 2024-09-11T05:51:41.605 INFO:teuthology.orchestra.run.smithi165.stdout:2709 punch from 0x81d6819 to 0x81e4ff1, (0xe7d8 bytes) 2024-09-11T05:51:41.605 INFO:teuthology.orchestra.run.smithi165.stdout:2711 write 0x99d7c1f thru 0x99e7495 (0xf877 bytes) 2024-09-11T05:51:41.610 INFO:teuthology.orchestra.run.smithi165.stdout:2712 read 0x3b03fd2 thru 0x3b07e96 (0x3ec5 bytes) 2024-09-11T05:51:41.611 INFO:teuthology.orchestra.run.smithi165.stdout:2713 write 0x40bf21d thru 0x40c8984 (0x9768 bytes) 2024-09-11T05:51:41.614 INFO:teuthology.orchestra.run.smithi165.stdout:2714 punch from 0xcd91cb1 to 0xcd91e67, (0x1b6 bytes) 2024-09-11T05:51:41.614 INFO:teuthology.orchestra.run.smithi165.stdout:2716 writesame 0x63ece8f thru 0x63fbfe7 data_size 0xf(0xf159 bytes) 2024-09-11T05:51:41.619 INFO:teuthology.orchestra.run.smithi165.stdout:2717 write 0xb92abb8 thru 0xb933cd2 (0x911b bytes) 2024-09-11T05:51:41.622 INFO:teuthology.orchestra.run.smithi165.stdout:2718 punch from 0x833e2bf to 0x833f947, (0x1688 bytes) 2024-09-11T05:51:41.633 INFO:teuthology.orchestra.run.smithi165.stdout:2720 trunc from 0xdc1b635 to 0xa210635 2024-09-11T05:51:41.661 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:41.660+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:41.676 INFO:teuthology.orchestra.run.smithi165.stdout:2721 write 0xcf924a9 thru 0xcf9bfc7 (0x9b1f bytes) 2024-09-11T05:51:41.682 INFO:teuthology.orchestra.run.smithi165.stdout:2725 read 0x5d12792 thru 0x5d1b135 (0x89a4 bytes) 2024-09-11T05:51:41.683 INFO:teuthology.orchestra.run.smithi165.stdout:2726 read 0x87d135f thru 0x87e0320 (0xefc2 bytes) 2024-09-11T05:51:41.684 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:41.685 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:41.685 INFO:teuthology.orchestra.run.smithi165.stdout:2729 punch from 0xb2bb45f to 0xb2c159e, (0x613f bytes) 2024-09-11T05:51:41.685 INFO:teuthology.orchestra.run.smithi165.stdout:2730 write 0xc3e2632 thru 0xc3ea395 (0x7d64 bytes) 2024-09-11T05:51:41.689 INFO:teuthology.orchestra.run.smithi165.stdout:2732 write 0xb4a24b3 thru 0xb4a59e0 (0x352e bytes) 2024-09-11T05:51:41.693 INFO:teuthology.orchestra.run.smithi165.stdout:2733 writesame 0x6eb5e20 thru 0x6ec0a50 data_size 0x1(0xac31 bytes) 2024-09-11T05:51:41.729 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:41.725+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:41.739 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:41.739 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:41.740 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:41.740 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:41.748 INFO:teuthology.orchestra.run.smithi165.stdout:2734 write 0x99f2b47 thru 0x99f49ca (0x1e84 bytes) 2024-09-11T05:51:41.752 INFO:teuthology.orchestra.run.smithi165.stdout:2735 read 0x6fcadac thru 0x6fd786a (0xcabf bytes) 2024-09-11T05:51:41.753 INFO:teuthology.orchestra.run.smithi165.stdout:2737 write 0xb7902a9 thru 0xb79433c (0x4094 bytes) 2024-09-11T05:51:41.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:41.757 INFO:teuthology.orchestra.run.smithi165.stdout:2738 write 0xcb97f43 thru 0xcba01c0 (0x827e bytes) 2024-09-11T05:51:41.764 INFO:teuthology.orchestra.run.smithi165.stdout:2740 read 0xcc6785c thru 0xcc74416 (0xcbbb bytes) 2024-09-11T05:51:41.764 INFO:teuthology.orchestra.run.smithi165.stdout:2741 read 0x24aa779 thru 0x24b9850 (0xf0d8 bytes) 2024-09-11T05:51:41.765 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:41.765 INFO:teuthology.orchestra.run.smithi165.stdout:2743 write 0x5a1096c thru 0x5a1b13f (0xa7d4 bytes) 2024-09-11T05:51:41.768 INFO:teuthology.orchestra.run.smithi165.stdout:2744 read 0x53b6573 thru 0x53c3a93 (0xd521 bytes) 2024-09-11T05:51:41.769 INFO:teuthology.orchestra.run.smithi165.stdout:2745 read 0xa476028 thru 0xa47cbaa (0x6b83 bytes) 2024-09-11T05:51:41.770 INFO:teuthology.orchestra.run.smithi165.stdout:2746 punch from 0xce30d77 to 0xce3daee, (0xcd77 bytes) 2024-09-11T05:51:41.770 INFO:teuthology.orchestra.run.smithi165.stdout:2747 write 0x25cd288 thru 0x25d2376 (0x50ef bytes) 2024-09-11T05:51:41.772 INFO:teuthology.orchestra.run.smithi165.stdout:2750 write 0xc93a68b thru 0xc93c5a1 (0x1f17 bytes) 2024-09-11T05:51:41.775 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.1 to 0.29087 (4a76) 2024-09-11T05:51:41.775 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:51:41.783 INFO:teuthology.orchestra.run.smithi165.stdout:2751 read 0x381708e thru 0x381ab4e (0x3ac1 bytes) 2024-09-11T05:51:41.841 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:41.840+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:41.920 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:41.920 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:41.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.161038687000001, 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:41.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:41.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:41.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.926 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.1178105739999999, 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:41.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:41.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:41.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68534 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.324274543, 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030432710000000002, 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.933 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68534 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:41.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594460+0000", 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6700000000000003e-07 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594463+0000", 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1420000000000002e-06 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594469+0000", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5150000000000003e-06 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594475+0000", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.003e-06 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624737+0000", 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030261957999999999 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624807+0000", 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9529999999999993e-05 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624892+0000", 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4795000000000004e-05 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.324249338, 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:41.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:51:41.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:51:41.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3240868459999999, 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:51:41.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3240617750000001, 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.946 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:41.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:51:41.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:51:41.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3240376979999999, 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.950 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3239870149999999, 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.323957415, 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.958 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:51:41.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:51:41.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3239317939999999, 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.962 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.966 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3239064149999999, 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:41.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:51:41.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:51:41.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3238789049999999, 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.323849788, 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:41.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:51:41.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:51:41.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3238235249999999, 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:51:41.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:51:41.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:51:41.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:51:41.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3236290660000001, 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.983 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3236059440000001, 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.987 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:41.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:51:41.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:51:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3235789659999999, 2024-09-11T05:51:41.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:41.995 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.323553207, 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:41.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:41.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:41.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.3235261149999999, 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:51:42.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:51:42.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:51:42.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:51:42.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.323502271, 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:42.004 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:42.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:42.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:42.008 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:42.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:42.010 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:42.008+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:42.110 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:42.105+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:42.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:42.181 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:42.182 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:42.182 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:42.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:42.219 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:42.217+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:42.311 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:42.308+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:42.318 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:42.318 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:42.318 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:42.318 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:42.318 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:42.331 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:42.331 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:42.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:42.347 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:42.347 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:42.366 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:42.366 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:42.384 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:42.384 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:42.402 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:42.402 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:42.412 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:42.409+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:42.420 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:42.420 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:42.438 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:42.439 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:42.457 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:42.458 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:42.495 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:42.495 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:42.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:42.516 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:42.513+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:42.613 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:42.609+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:42.626 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:42.626 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:42.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:42.714 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:42.709+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:42.759 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:42.759 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:42.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:42.815 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:42.809+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:42.901 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:42.901 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:42.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:42.916 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:42.913+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:43.016 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:43.013+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:43.045 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.045 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:43.144 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:43.141+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:43.218 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:43.213+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:43.222 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.222 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:43.323 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:43.321+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:43.399 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.400 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:43.420 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:43.420+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:43.521 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:43.520+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:43.580 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.581 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:43.639 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:43.637+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:43.716 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.717 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.722 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:43.717+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:43.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:43.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:51:43.823 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:43.820+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:43.907 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:43.907 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:43.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:43.924 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:43.920+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:44.025 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:44.021+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:44.042 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:44.042 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:44.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:44.126 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:44.124+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:44.221 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:44.222 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:44.227 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:44.224+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:44.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:44.241 INFO:teuthology.orchestra.run.smithi165.stdout:2752 writesame 0x4860f7c thru 0x486b690 data_size 0x1(0xa715 bytes) 2024-09-11T05:51:44.267 INFO:teuthology.orchestra.run.smithi165.stdout:2754 read 0x8608c72 thru 0x861303c (0xa3cb bytes) 2024-09-11T05:51:44.268 INFO:teuthology.orchestra.run.smithi165.stdout:2755 read 0x669e139 thru 0x66a6cbd (0x8b85 bytes) 2024-09-11T05:51:44.269 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.269 INFO:teuthology.orchestra.run.smithi165.stdout:2758 write 0x9499783 thru 0x94a8ce2 (0xf560 bytes) 2024-09-11T05:51:44.293 INFO:teuthology.orchestra.run.smithi165.stdout:2759 trunc from 0xcf9bfc8 to 0x48d2cf0 2024-09-11T05:51:44.328 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:44.324+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:44.331 INFO:teuthology.orchestra.run.smithi165.stdout:2760 read 0x167763f thru 0x167c975 (0x5337 bytes) 2024-09-11T05:51:44.332 INFO:teuthology.orchestra.run.smithi165.stdout:2761 writesame 0x467eab8 thru 0x4689167 data_size 0x1c(0xa6b0 bytes) 2024-09-11T05:51:44.336 INFO:teuthology.orchestra.run.smithi165.stdout:2762 punch from 0xe36b13 to 0xe3c8f7, (0x5de4 bytes) 2024-09-11T05:51:44.348 INFO:teuthology.orchestra.run.smithi165.stdout:2763 trunc from 0x48d2cf0 to 0xc12cbc3 2024-09-11T05:51:44.362 INFO:teuthology.orchestra.run.smithi165.stdout:2764 read 0x76a8969 thru 0x76ac5f5 (0x3c8d bytes) 2024-09-11T05:51:44.362 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.362 INFO:teuthology.orchestra.run.smithi165.stdout:2766 write 0x709d3c3 thru 0x70a4b3a (0x7778 bytes) 2024-09-11T05:51:44.385 INFO:teuthology.orchestra.run.smithi165.stdout:2767 trunc from 0xc12cbc3 to 0x842fcc 2024-09-11T05:51:44.396 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:44.396 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:44.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:44.413 INFO:teuthology.orchestra.run.smithi165.stdout:2769 trunc from 0x842fcc to 0x1727d88 2024-09-11T05:51:44.425 INFO:teuthology.orchestra.run.smithi165.stdout:2771 read 0xc1ede5 thru 0xc21a4e (0x2c6a bytes) 2024-09-11T05:51:44.429 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:44.428+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:44.453 INFO:teuthology.orchestra.run.smithi165.stdout:2772 write 0x9a825d1 thru 0x9a8c181 (0x9bb1 bytes) 2024-09-11T05:51:44.458 INFO:teuthology.orchestra.run.smithi165.stdout:2774 punch from 0x547842a to 0x5482974, (0xa54a bytes) 2024-09-11T05:51:44.476 INFO:teuthology.orchestra.run.smithi165.stdout:2775 writesame 0xaa4b046 thru 0xaa4b0ed data_size 0x3(0xa8 bytes) 2024-09-11T05:51:44.477 INFO:teuthology.orchestra.run.smithi165.stdout:2776 read 0x6debd2a thru 0x6df3e9f (0x8176 bytes) 2024-09-11T05:51:44.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:44.481 INFO:teuthology.orchestra.run.smithi165.stdout:2778 trunc from 0xaa4b0ee to 0xd2e50c4 2024-09-11T05:51:44.494 INFO:teuthology.orchestra.run.smithi165.stdout:2780 writesame 0x3e85268 thru 0x3e8e8be data_size 0x1(0x9657 bytes) 2024-09-11T05:51:44.525 INFO:teuthology.orchestra.run.smithi165.stdout:2781 write 0x23a4fc1 thru 0x23ad350 (0x8390 bytes) 2024-09-11T05:51:44.530 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:44.525+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:44.533 INFO:teuthology.orchestra.run.smithi165.stdout:2782 trunc from 0xd2e50c4 to 0x9f3ecd6 2024-09-11T05:51:44.546 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:44.547 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:44.553 INFO:teuthology.orchestra.run.smithi165.stdout:2784 punch from 0x1e8b2c5 to 0x1e8f720, (0x445b bytes) 2024-09-11T05:51:44.553 INFO:teuthology.orchestra.run.smithi165.stdout:2785 writesame 0x5607ada thru 0x5609ca5 data_size 0x1c(0x21cc bytes) 2024-09-11T05:51:44.556 INFO:teuthology.orchestra.run.smithi165.stdout:2786 read 0x5a38597 thru 0x5a4155f (0x8fc9 bytes) 2024-09-11T05:51:44.557 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.557 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.557 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.571 INFO:teuthology.orchestra.run.smithi165.stdout:2791 trunc from 0x9f3ecd6 to 0x1808c8d 2024-09-11T05:51:44.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:44.591 INFO:teuthology.orchestra.run.smithi165.stdout:2792 punch from 0x1af628 to 0x1bb9a4, (0xc37c bytes) 2024-09-11T05:51:44.591 INFO:teuthology.orchestra.run.smithi165.stdout:2793 read 0x114dffc thru 0x1159832 (0xb837 bytes) 2024-09-11T05:51:44.592 INFO:teuthology.orchestra.run.smithi165.stdout:2794 punch from 0x413275 to 0x41a6b3, (0x743e bytes) 2024-09-11T05:51:44.614 INFO:teuthology.orchestra.run.smithi165.stdout:2795 writesame 0x8f13ad1 thru 0x8f1660f data_size 0x1(0x2b3f bytes) 2024-09-11T05:51:44.631 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:44.625+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:44.632 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:44.632 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:44.638 INFO:teuthology.orchestra.run.smithi165.stdout:2796 write 0xdfed4ba thru 0xdffbdca (0xe911 bytes) 2024-09-11T05:51:44.644 INFO:teuthology.orchestra.run.smithi165.stdout:2797 writesame 0x7a31aee thru 0x7a33373 data_size 0x2(0x1886 bytes) 2024-09-11T05:51:44.648 INFO:teuthology.orchestra.run.smithi165.stdout:2798 write 0x46ea620 thru 0x46eec6b (0x464c bytes) 2024-09-11T05:51:44.650 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:44.650 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:44.652 INFO:teuthology.orchestra.run.smithi165.stdout:2799 read 0x80677f9 thru 0x80709b1 (0x91b9 bytes) 2024-09-11T05:51:44.653 INFO:teuthology.orchestra.run.smithi165.stdout:2801 write 0x97f57cb thru 0x9801b73 (0xc3a9 bytes) 2024-09-11T05:51:44.657 INFO:teuthology.orchestra.run.smithi165.stdout:2802 read 0x632f79e thru 0x6333901 (0x4164 bytes) 2024-09-11T05:51:44.658 INFO:teuthology.orchestra.run.smithi165.stdout:2803 write 0xa4e1e0a thru 0xa4e93e6 (0x75dd bytes) 2024-09-11T05:51:44.662 INFO:teuthology.orchestra.run.smithi165.stdout:2804 writesame 0xbfd341d thru 0xbfdac8e data_size 0x1(0x7872 bytes) 2024-09-11T05:51:44.667 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:44.667 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:44.684 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:44.684 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:44.706 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:44.706 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:44.712 INFO:teuthology.orchestra.run.smithi165.stdout:2805 write 0xe218292 thru 0xe2224f3 (0xa262 bytes) 2024-09-11T05:51:44.716 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:44.716 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:44.716 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:51:44.716 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:51:44.717 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:44.728 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:44.728 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:44.731 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:44.728+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:44.739 INFO:teuthology.orchestra.run.smithi165.stdout:2806 trunc from 0xe2224f4 to 0x2caac81 2024-09-11T05:51:44.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:44.750 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:44.750 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:44.774 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:44.774 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:44.777 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:44.794 INFO:teuthology.orchestra.run.smithi165.stdout:2809 writesame 0x5f8056c thru 0x5f82bf1 data_size 0x1(0x2686 bytes) 2024-09-11T05:51:44.796 INFO:teuthology.orchestra.run.smithi165.stdout:2813 read 0x3a07775 thru 0x3a15f2c (0xe7b8 bytes) 2024-09-11T05:51:44.796 INFO:teuthology.orchestra.run.smithi165.stdout:2814 write 0x434894d thru 0x43554ee (0xcba2 bytes) 2024-09-11T05:51:44.800 INFO:teuthology.orchestra.run.smithi165.stdout:2818 write 0x1870418 thru 0x187cd2f (0xc918 bytes) 2024-09-11T05:51:44.803 INFO:teuthology.orchestra.run.smithi165.stdout:2820 punch from 0x5dd368c to 0x5de11f7, (0xdb6b bytes) 2024-09-11T05:51:44.803 INFO:teuthology.orchestra.run.smithi165.stdout:2821 punch from 0xb1596b to 0xb1a3c2, (0x4a57 bytes) 2024-09-11T05:51:44.803 INFO:teuthology.orchestra.run.smithi165.stdout:2822 write 0x3535ae5 thru 0x353cf9a (0x74b6 bytes) 2024-09-11T05:51:44.809 INFO:teuthology.orchestra.run.smithi165.stdout:2823 trunc from 0x5f82bf2 to 0x4564b17 2024-09-11T05:51:44.822 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:51:44.832 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:44.829+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:44.840 INFO:teuthology.orchestra.run.smithi165.stdout:2824 read 0x3db2267 thru 0x3db55f4 (0x338e bytes) 2024-09-11T05:51:44.877 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:44.877 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73287 3.8 3:17ede237:::rbd_data.1124767f15a5.0000000000000042:12 [read 385024~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071092+0000", 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8064089220000001, 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033766589999999999, 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.878 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73287 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071092+0000", 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071092+0000", 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071092+0000", 2024-09-11T05:51:44.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3500000000000002e-07 2024-09-11T05:51:44.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071097+0000", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8829999999999997e-06 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071103+0000", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0750000000000003e-06 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071108+0000", 2024-09-11T05:51:44.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5600000000000004e-06 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104366+0000", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03325819 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104438+0000", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2390000000000003e-05 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104858+0000", 2024-09-11T05:51:44.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000420157 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73336 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1331200~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8063324889999999, 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034169613000000001, 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73336 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071169+0000", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4799999999999999e-07 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071170+0000", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.821e-06 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071176+0000", 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8900000000000004e-06 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071181+0000", 2024-09-11T05:51:44.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4469999999999999e-06 2024-09-11T05:51:44.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104776+0000", 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033594945000000001 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104844+0000", 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8405999999999995e-05 2024-09-11T05:51:44.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105338+0000", 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049375600000000003 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73347 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1361920~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8063142599999997, 2024-09-11T05:51:44.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034640307000000002, 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73347 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071187+0000", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071189+0000", 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.843e-06 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071195+0000", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071199+0000", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6410000000000001e-06 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105705+0000", 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034505909000000001 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105768+0000", 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2871000000000002e-05 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105827+0000", 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8795000000000002e-05 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.893 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73350 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1576960~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.806295913, 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034310473000000001, 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73350 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5999999999999999e-07 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071207+0000", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.832e-06 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071213+0000", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7409999999999998e-06 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071217+0000", 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4989999999999996e-06 2024-09-11T05:51:44.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105375+0000", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034157869 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105440+0000", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5056999999999997e-05 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105515+0000", 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5115000000000002e-05 2024-09-11T05:51:44.897 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73352 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1607680~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8062779020000002, 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034453226000000003, 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73352 2024-09-11T05:51:44.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071225+0000", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071231+0000", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8309999999999997e-06 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071235+0000", 2024-09-11T05:51:44.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.476e-06 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105538+0000", 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034302323000000003 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105603+0000", 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4969999999999996e-05 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105676+0000", 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3435000000000004e-05 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73353 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1581056~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8062596549999999, 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035152165999999999, 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.902 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73353 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071243+0000", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071249+0000", 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6869999999999999e-06 2024-09-11T05:51:44.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071253+0000", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106025+0000", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034771560999999999 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106088+0000", 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2911999999999997e-05 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106393+0000", 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000305352 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73360 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 69632~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.806241403, 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034953914000000003, 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73360 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071260+0000", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7599999999999998e-07 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071262+0000", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3769999999999999e-06 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071269+0000", 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5130000000000002e-06 2024-09-11T05:51:44.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071273+0000", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8509999999999998e-06 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105850+0000", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034576140999999998 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105920+0000", 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0351000000000005e-05 2024-09-11T05:51:44.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106213+0000", 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000293305 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73361 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 278528~18432] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8062208320000002, 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035273718000000003, 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.910 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73361 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071282+0000", 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071289+0000", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3090000000000001e-06 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071293+0000", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7879999999999997e-06 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106234+0000", 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034940183999999999 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106301+0000", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7638000000000005e-05 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106554+0000", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025235999999999998 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73362 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 489472~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8062010819999998, 2024-09-11T05:51:44.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035455391000000003, 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73362 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0499999999999999e-07 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071302+0000", 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-06 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071308+0000", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071313+0000", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6299999999999997e-06 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106625+0000", 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035312398000000002 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106693+0000", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7954999999999996e-05 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106755+0000", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2055000000000001e-05 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73367 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1611776~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8061824500000001, 2024-09-11T05:51:44.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036019230999999999, 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73367 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071319+0000", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4400000000000001e-07 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071325+0000", 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071331+0000", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071335+0000", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6319999999999999e-06 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106578+0000", 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035242348999999999 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107279+0000", 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070118999999999997 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107337+0000", 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8646000000000001e-05 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73370 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1888256~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.922 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8061585010000001, 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035569649000000002, 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73370 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071343+0000", 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071344+0000", 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071350+0000", 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071355+0000", 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3850000000000004e-06 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106778+0000", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035423221999999997 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106840+0000", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106912+0000", 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2181000000000006e-05 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.926 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73372 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1329152~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8061402160000002, 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035712460000000001, 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73372 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.927 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071361+0000", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9499999999999998e-07 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071363+0000", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071369+0000", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071374+0000", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.78e-06 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106936+0000", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035561622000000001 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106998+0000", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1820000000000001e-05 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107073+0000", 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5210999999999995e-05 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73374 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1359872~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8061203519999998, 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035851052000000001, 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73374 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071381+0000", 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5499999999999999e-07 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071383+0000", 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071388+0000", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7370000000000003e-06 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071393+0000", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8069999999999999e-06 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107095+0000", 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035701644999999997 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107157+0000", 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107231+0000", 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4828999999999998e-05 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73375 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1333248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.935 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8061012669999998, 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036088839999999997, 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73375 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:51:44.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071400+0000", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071402+0000", 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.148e-06 2024-09-11T05:51:44.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071409+0000", 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5159999999999997e-06 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071414+0000", 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0000000000000004e-06 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107365+0000", 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035950332000000002 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107429+0000", 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4271999999999997e-05 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107488+0000", 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9215999999999997e-05 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.939 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73377 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1363968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8060797179999999, 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036963658000000003, 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73377 2024-09-11T05:51:44.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071424+0000", 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.419e-06 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071431+0000", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071437+0000", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0919999999999997e-06 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108262+0000", 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036825113999999999 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108323+0000", 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1292999999999997e-05 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108385+0000", 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1651999999999999e-05 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73380 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1579008~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8060572989999999, 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036798605999999998, 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73380 2024-09-11T05:51:44.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071444+0000", 2024-09-11T05:51:44.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071446+0000", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2929999999999999e-06 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071453+0000", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0049999999999998e-06 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071458+0000", 2024-09-11T05:51:44.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-06 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107511+0000", 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036052646000000001 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108165+0000", 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000653934 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108242+0000", 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7222e-05 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73382 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1609728~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8060356999999998, 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037077023000000001, 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.949 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73382 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.65e-07 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071468+0000", 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:51:44.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071475+0000", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071480+0000", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2020000000000003e-06 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108408+0000", 2024-09-11T05:51:44.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036928043000000001 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108470+0000", 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1865999999999993e-05 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108542+0000", 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2206999999999995e-05 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73383 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1583104~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8060140379999998, 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037204763000000002, 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73383 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071489+0000", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.373e-06 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071495+0000", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8250000000000001e-06 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071500+0000", 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108564+0000", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037063974 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108628+0000", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3634999999999996e-05 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108691+0000", 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3647999999999998e-05 2024-09-11T05:51:44.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73385 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1613824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8059940450000003, 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037327601000000002, 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73385 2024-09-11T05:51:44.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8299999999999998e-07 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071509+0000", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0829999999999998e-06 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071517+0000", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071523+0000", 2024-09-11T05:51:44.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9179999999999999e-06 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108714+0000", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037191175999999999 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108775+0000", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108834+0000", 2024-09-11T05:51:44.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9447999999999997e-05 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73390 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1890304~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.8059712060000002, 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037487624999999997, 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:51:44.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73390 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071530+0000", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0900000000000002e-07 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071532+0000", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071538+0000", 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071543+0000", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9930000000000003e-06 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108856+0000", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037312788 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108916+0000", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9570000000000001e-05 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:51:44.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.109017+0000", 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101388 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:44.966 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:44.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:44.968 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:44.965+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:45.068 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:45.065+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:45.098 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:45.098 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:45.098 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:45.099 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:45.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:45.169 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:45.168+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:45.242 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:45.242 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:45.242 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:45.242 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:45.242 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:45.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:45.270 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:45.265+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:45.371 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:45.368+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:45.425 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58950 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3450499819999999, 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064766338000000007, 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58950 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079620+0000", 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1880000000000001e-06 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079627+0000", 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2740000000000004e-06 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079633+0000", 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1770000000000003e-06 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144235+0000", 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064602407000000001 2024-09-11T05:51:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144301+0000", 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6125999999999997e-05 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144384+0000", 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2696999999999998e-05 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58951 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 966656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.345028117, 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064870423999999996, 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58951 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079642+0000", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2910000000000002e-06 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079649+0000", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999999e-06 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079653+0000", 2024-09-11T05:51:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9439999999999996e-06 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144413+0000", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064759540000000004 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144455+0000", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2438000000000003e-05 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144510+0000", 2024-09-11T05:51:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4573000000000001e-05 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58952 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3449986059999999, 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065131952000000007, 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58952 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079670+0000", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079672+0000", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.627e-06 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079680+0000", 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6590000000000007e-06 2024-09-11T05:51:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079685+0000", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3469999999999998e-06 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144541+0000", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064855876000000007 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144725+0000", 2024-09-11T05:51:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018391499999999999 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144801+0000", 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5826000000000002e-05 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58953 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.344974713, 2024-09-11T05:51:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065264442000000006, 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58953 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079694+0000", 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2799999999999995e-07 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079696+0000", 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0789999999999999e-06 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079702+0000", 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.455e-06 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079708+0000", 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7749999999999998e-06 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144832+0000", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065124390000000004 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144887+0000", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4669000000000001e-05 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144957+0000", 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0345999999999994e-05 2024-09-11T05:51:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58954 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3449530740000002, 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066242835, 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58954 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.08e-07 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079723+0000", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2830000000000001e-06 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079730+0000", 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6059999999999999e-06 2024-09-11T05:51:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079735+0000", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1930000000000001e-06 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145779+0000", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066043973000000006 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145861+0000", 2024-09-11T05:51:45.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1984000000000007e-05 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145957+0000", 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6387999999999999e-05 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58959 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2015232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:45.448 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3449245190000001, 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065383153999999999, 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58959 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:45.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079744+0000", 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079746+0000", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4830000000000002e-06 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079759+0000", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3113e-05 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079766+0000", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.832e-06 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144985+0000", 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065218999999999999 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145060+0000", 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5186999999999997e-05 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145126+0000", 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6126999999999999e-05 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58966 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2093056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.344893882, 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065511660999999999, 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58966 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999996e-07 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079777+0000", 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6740000000000001e-06 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079782+0000", 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7559999999999999e-06 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079787+0000", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3580000000000002e-06 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145153+0000", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065365728999999997 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145219+0000", 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5720000000000001e-05 2024-09-11T05:51:45.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145285+0000", 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6927000000000005e-05 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58972 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2109440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3448757249999996, 2024-09-11T05:51:45.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065642100999999994, 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58972 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:45.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080613+0000", 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00082077599999999995 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080617+0000", 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080627+0000", 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.004e-05 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080634+0000", 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7519999999999999e-06 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145311+0000", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064677068000000004 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145367+0000", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6137999999999997e-05 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145434+0000", 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6931e-05 2024-09-11T05:51:45.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58978 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2187264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3440111840000002, 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064940567000000005, 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.462 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58978 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:45.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080660+0000", 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8629999999999999e-06 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080668+0000", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0879999999999995e-06 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080682+0000", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3788000000000001e-05 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145461+0000", 2024-09-11T05:51:45.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064779310000000007 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145528+0000", 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6764e-05 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145597+0000", 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9052000000000002e-05 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58985 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2234368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3439757820000002, 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065076868999999996, 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58985 2024-09-11T05:51:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080695+0000", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4959999999999999e-06 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080702+0000", 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3529999999999997e-06 2024-09-11T05:51:45.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080708+0000", 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145624+0000", 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064915918000000003 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145693+0000", 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8759999999999999e-05 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145769+0000", 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5862e-05 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58991 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2312192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3439522469999998, 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065155451000000003, 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58991 2024-09-11T05:51:45.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080716+0000", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2500000000000001e-07 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080718+0000", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080725+0000", 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:51:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080730+0000", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2989999999999996e-06 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145793+0000", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065062732999999998 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145831+0000", 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8136999999999998e-05 2024-09-11T05:51:45.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145871+0000", 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9745e-05 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58997 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3439307740000004, 2024-09-11T05:51:45.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065296924000000006, 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58997 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080837+0000", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9950000000000004e-05 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080841+0000", 2024-09-11T05:51:45.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2610000000000004e-06 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080850+0000", 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5639999999999997e-06 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080856+0000", 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.669e-06 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145895+0000", 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065038809000000003 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145956+0000", 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0461999999999998e-05 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146034+0000", 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8208999999999996e-05 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58998 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2390016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3438003900000002, 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065346670999999995, 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58998 2024-09-11T05:51:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081070+0000", 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020277699999999999 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081075+0000", 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.972e-06 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081085+0000", 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9459999999999993e-06 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081093+0000", 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9060000000000002e-06 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146084+0000", 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064990669000000001 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146155+0000", 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1252000000000002e-05 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146214+0000", 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9148999999999999e-05 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59006 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2140160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:45.485 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3435603880000002, 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065251679000000007, 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59006 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:45.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081305+0000", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197519 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081309+0000", 2024-09-11T05:51:45.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081318+0000", 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7900000000000005e-06 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081325+0000", 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4950000000000002e-06 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146244+0000", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064918645999999997 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146303+0000", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8668000000000001e-05 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146359+0000", 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6471999999999997e-05 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.489 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59012 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2217984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3433242959999996, 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065189569000000003, 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.490 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59012 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082116+0000", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077256999999999998 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082144+0000", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7863000000000001e-05 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082163+0000", 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9378999999999999e-05 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082195+0000", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1782999999999997e-05 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146393+0000", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064197562999999999 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146460+0000", 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7197000000000001e-05 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146533+0000", 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3213999999999994e-05 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59024 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2342912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3424319049999998, 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064656276999999998, 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59024 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:45.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082244+0000", 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1999999999999994e-06 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082252+0000", 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.435e-06 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082258+0000", 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8610000000000003e-06 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146738+0000", 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064479936000000002 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146814+0000", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6728000000000001e-05 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146892+0000", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7645999999999995e-05 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59031 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2420736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3424018909999997, 2024-09-11T05:51:45.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064743414999999999, 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59031 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.63e-07 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082269+0000", 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2670000000000001e-06 2024-09-11T05:51:45.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082275+0000", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2759999999999997e-06 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082280+0000", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9269999999999996e-06 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146928+0000", 2024-09-11T05:51:45.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064647901999999993 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146975+0000", 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6860000000000002e-05 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147009+0000", 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.472e-05 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59039 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2437120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3423813510000002, 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848283000000007, 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59039 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.503 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082342+0000", 2024-09-11T05:51:45.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5473000000000002e-05 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082346+0000", 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8310000000000003e-06 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082354+0000", 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6549999999999993e-06 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082361+0000", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147025+0000", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064663913000000003 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147082+0000", 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6805000000000002e-05 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147135+0000", 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2438999999999997e-05 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59045 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2514944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3422959739999998, 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064917090999999996, 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.507 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59045 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083064+0000", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069213100000000002 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083069+0000", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2789999999999998e-06 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083081+0000", 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2251e-05 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083089+0000", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8140000000000001e-06 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147163+0000", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064073571999999995 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147228+0000", 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5235999999999997e-05 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147289+0000", 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0807999999999998e-05 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59068 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2592768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3413022239999997, 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064977262999999993, 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:45.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59068 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:45.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.084910+0000", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001544836 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085422+0000", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051133000000000005 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085433+0000", 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085441+0000", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179999999999998e-06 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148208+0000", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062766695999999997 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148273+0000", 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5395999999999995e-05 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:45.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148343+0000", 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9807000000000004e-05 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:45.515 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:45.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:45.517 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:45.513+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:45.618 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:45.616+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:45.689 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:45.689 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:45.689 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:45.689 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:45.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:45.719 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:45.717+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:45.819 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:45.817+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:45.841 INFO:teuthology.orchestra.run.smithi165.stdout:2825 read 0xb3a4ae thru 0xb47e51 (0xd9a4 bytes) 2024-09-11T05:51:45.842 INFO:teuthology.orchestra.run.smithi165.stdout:2827 write 0x2729468 thru 0x2729af7 (0x690 bytes) 2024-09-11T05:51:45.848 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:45.848 INFO:teuthology.orchestra.run.smithi165.stdout:2829 punch from 0x398d0a1 to 0x399211f, (0x507e bytes) 2024-09-11T05:51:45.848 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:45.849 INFO:teuthology.orchestra.run.smithi165.stdout:2833 read 0x190baef thru 0x1919da8 (0xe2ba bytes) 2024-09-11T05:51:45.878 INFO:teuthology.orchestra.run.smithi165.stdout:2834 write 0x8b9aa95 thru 0x8b9f136 (0x46a2 bytes) 2024-09-11T05:51:45.879 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:45.879 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:45.879 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:45.879 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:45.879 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:45.881 INFO:teuthology.orchestra.run.smithi165.stdout:2838 read 0x499cb09 thru 0x49a8249 (0xb741 bytes) 2024-09-11T05:51:45.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:45.903 INFO:teuthology.orchestra.run.smithi165.stdout:2839 write 0xb65ad4b thru 0xb66042b (0x56e1 bytes) 2024-09-11T05:51:45.907 INFO:teuthology.orchestra.run.smithi165.stdout:2840 punch from 0x56fdf75 to 0x5700909, (0x2994 bytes) 2024-09-11T05:51:45.907 INFO:teuthology.orchestra.run.smithi165.stdout:2841 punch from 0x75d9718 to 0x75e4659, (0xaf41 bytes) 2024-09-11T05:51:45.907 INFO:teuthology.orchestra.run.smithi165.stdout:2843 read 0x43dad7c thru 0x43e50e3 (0xa368 bytes) 2024-09-11T05:51:45.924 INFO:teuthology.orchestra.run.smithi165.stdout:2844 write 0xe0167af thru 0xe01cc3f (0x6491 bytes) 2024-09-11T05:51:45.929 INFO:teuthology.orchestra.run.smithi165.stdout:2845 punch from 0x5966d81 to 0x59729c9, (0xbc48 bytes) 2024-09-11T05:51:45.929 INFO:teuthology.orchestra.run.smithi165.stdout:2846 writesame 0x7ad5dd4 thru 0x7ae1b18 data_size 0x1f(0xbd45 bytes) 2024-09-11T05:51:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:2848 write 0x599267c thru 0x599988a (0x720f bytes) 2024-09-11T05:51:45.937 INFO:teuthology.orchestra.run.smithi165.stdout:2849 read 0x24fee43 thru 0x2505c84 (0x6e42 bytes) 2024-09-11T05:51:45.938 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:45.941 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:45.940+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:45.966 INFO:teuthology.orchestra.run.smithi165.stdout:2852 trunc from 0xe01cc40 to 0x373c11e 2024-09-11T05:51:45.999 INFO:teuthology.orchestra.run.smithi165.stdout:2853 write 0x3a1387 thru 0x3a954d (0x81c7 bytes) 2024-09-11T05:51:46.022 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:46.020+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.266850852000001, 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.223622739, 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68534 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4300867080000002, 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030432710000000002, 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68534 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594460+0000", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6700000000000003e-07 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594463+0000", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1420000000000002e-06 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594469+0000", 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5150000000000003e-06 2024-09-11T05:51:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594475+0000", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.003e-06 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624737+0000", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030261957999999999 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624807+0000", 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9529999999999993e-05 2024-09-11T05:51:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624892+0000", 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4795000000000004e-05 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4300615030000001, 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:51:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4298990109999998, 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:51:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:51:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4298739400000002, 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:51:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:51:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:51:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4298498630000003, 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4297991799999998, 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:51:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:51:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4297695800000003, 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:46.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4297439589999996, 2024-09-11T05:51:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4297185800000003, 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:51:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:51:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4296910699999996, 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4296619530000001, 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4296356899999996, 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:51:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:51:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:51:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:51:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4294412310000002, 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4294181090000002, 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.429391131, 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4293653720000004, 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4293382799999996, 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:51:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:51:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.4293144360000003, 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.113 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:46.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2854 write 0xb13a8a9 thru 0xb149063 (0xe7bb bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2856 write 0xe264cbf thru 0xe268b16 (0x3e58 bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2857 read 0x56668fb thru 0x5668d9e (0x24a4 bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2858 write 0x972d1ad thru 0x97384ce (0xb322 bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2859 write 0x71bf9cf thru 0x71c27ba (0x2dec bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2861 write 0x985596a thru 0x985fb7c (0xa213 bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2862 writesame 0xa7a8fe4 thru 0xa7aae52 data_size 0x31(0x1e6f bytes) 2024-09-11T05:51:46.115 INFO:teuthology.orchestra.run.smithi165.stdout:2863 writesame 0xa96cffd thru 0xa976493 data_size 0x1(0x9497 bytes) 2024-09-11T05:51:46.123 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:46.117+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:46.135 INFO:teuthology.orchestra.run.smithi165.stdout:2865 read 0x48a87ae thru 0x48b68c3 (0xe116 bytes) 2024-09-11T05:51:46.136 INFO:teuthology.orchestra.run.smithi165.stdout:2866 writesame 0x24273a1 thru 0x2431f67 data_size 0x5(0xabc7 bytes) 2024-09-11T05:51:46.150 INFO:teuthology.orchestra.run.smithi165.stdout:2869 punch from 0x2f8692b to 0x2f8987b, (0x2f50 bytes) 2024-09-11T05:51:46.150 INFO:teuthology.orchestra.run.smithi165.stdout:2870 write 0x959108c thru 0x959fa7f (0xe9f4 bytes) 2024-09-11T05:51:46.156 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.156 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.156 INFO:teuthology.orchestra.run.smithi165.stdout:2875 read 0xb7a5ef8 thru 0xb7a7eee (0x1ff7 bytes) 2024-09-11T05:51:46.157 INFO:teuthology.orchestra.run.smithi165.stdout:2878 read 0xbdf6720 thru 0xbdf8bfc (0x24dd bytes) 2024-09-11T05:51:46.157 INFO:teuthology.orchestra.run.smithi165.stdout:2879 punch from 0x82c468b to 0x82ce772, (0xa0e7 bytes) 2024-09-11T05:51:46.191 INFO:teuthology.orchestra.run.smithi165.stdout:2880 trunc from 0xe268b17 to 0xe75063 2024-09-11T05:51:46.216 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.223 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:46.220+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:46.232 INFO:teuthology.orchestra.run.smithi165.stdout:2882 write 0x3588800 thru 0x358b0d6 (0x28d7 bytes) 2024-09-11T05:51:46.235 INFO:teuthology.orchestra.run.smithi165.stdout:2884 read 0xb172e1 thru 0xb256b9 (0xe3d9 bytes) 2024-09-11T05:51:46.236 INFO:teuthology.orchestra.run.smithi165.stdout:2885 read 0x1f46804 thru 0x1f4c3df (0x5bdc bytes) 2024-09-11T05:51:46.266 INFO:teuthology.orchestra.run.smithi165.stdout:2887 write 0x94e8402 thru 0x94f4948 (0xc547 bytes) 2024-09-11T05:51:46.270 INFO:teuthology.orchestra.run.smithi165.stdout:2888 read 0x7840ce0 thru 0x784ea38 (0xdd59 bytes) 2024-09-11T05:51:46.293 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:46.293 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:46.294 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:46.294 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:46.296 INFO:teuthology.orchestra.run.smithi165.stdout:2890 writesame 0xc7fd2f7 thru 0xc80b846 data_size 0x18(0xe550 bytes) 2024-09-11T05:51:46.296 INFO:teuthology.orchestra.run.smithi165.stdout:2891 write 0x5e85d49 thru 0x5e8666e (0x926 bytes) 2024-09-11T05:51:46.301 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.301 INFO:teuthology.orchestra.run.smithi165.stdout:2895 writesame 0x3b0dd51 thru 0x3b16f70 data_size 0x70(0x9220 bytes) 2024-09-11T05:51:46.306 INFO:teuthology.orchestra.run.smithi165.stdout:2896 read 0x8a1e5dd thru 0x8a1f477 (0xe9b bytes) 2024-09-11T05:51:46.307 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.307 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.307 INFO:teuthology.orchestra.run.smithi165.stdout:2900 writesame 0xa4c45ff thru 0xa4cfdca data_size 0x12(0xb7cc bytes) 2024-09-11T05:51:46.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:46.311 INFO:teuthology.orchestra.run.smithi165.stdout:2901 write 0x7347d2b thru 0x73515e9 (0x98bf bytes) 2024-09-11T05:51:46.316 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.317 INFO:teuthology.orchestra.run.smithi165.stdout:2903 write 0x66cf64e thru 0x66deb33 (0xf4e6 bytes) 2024-09-11T05:51:46.320 INFO:teuthology.orchestra.run.smithi165.stdout:2904 read 0x27302fd thru 0x2734bb3 (0x48b7 bytes) 2024-09-11T05:51:46.321 INFO:teuthology.orchestra.run.smithi165.stdout:2907 write 0xbf96bdd thru 0xbfa33d5 (0xc7f9 bytes) 2024-09-11T05:51:46.324 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:46.321+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:46.325 INFO:teuthology.orchestra.run.smithi165.stdout:2908 write 0x268a770 thru 0x26976bc (0xcf4d bytes) 2024-09-11T05:51:46.329 INFO:teuthology.orchestra.run.smithi165.stdout:2909 trunc from 0xc80b847 to 0xd114f49 2024-09-11T05:51:46.372 INFO:teuthology.orchestra.run.smithi165.stdout:2910 trunc from 0xd114f49 to 0x1722532 2024-09-11T05:51:46.388 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:46.424 INFO:teuthology.orchestra.run.smithi165.stdout:2913 writesame 0xbc36bd3 thru 0xbc3c575 data_size 0x1(0x59a3 bytes) 2024-09-11T05:51:46.425 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:46.421+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:46.427 INFO:teuthology.orchestra.run.smithi165.stdout:2914 punch from 0xef2998 to 0xefd5a8, (0xac10 bytes) 2024-09-11T05:51:46.427 INFO:teuthology.orchestra.run.smithi165.stdout:2916 punch from 0x7a2d55e to 0x7a30802, (0x32a4 bytes) 2024-09-11T05:51:46.434 INFO:teuthology.orchestra.run.smithi165.stdout:2917 trunc from 0xbc3c576 to 0xe69f4b8 2024-09-11T05:51:46.445 INFO:teuthology.orchestra.run.smithi165.stdout:2918 write 0x2d83879 thru 0x2d875a9 (0x3d31 bytes) 2024-09-11T05:51:46.448 INFO:teuthology.orchestra.run.smithi165.stdout:2919 clone 17 order 19 su 65536 sc 13 2024-09-11T05:51:46.483 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:46.483 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:46.483 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:46.483 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:46.483 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:46.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:46.525 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:46.521+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:46.624 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:46.624 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:1026 3.a e30/28 3:50a9d2c1:::rbd_data.112475afb12f.000000000000004c:head v 30'2, mlcod=0'0)", 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.772223240000002, 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043959489999999997, 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851063+0000", 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899999999999999e-06 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851141+0000", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851182+0000", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1207e-05 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851204+0000", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1812999999999999e-05 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851341+0000", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013705600000000001 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851391+0000", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9811999999999997e-05 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855349+0000", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00395825 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855456+0000", 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107582 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6402 3.a e33/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 33'21, mlcod=33'19)", 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.742017468, 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003674802, 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881269+0000", 2024-09-11T05:51:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6539999999999999e-06 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881325+0000", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609000000000002e-05 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881385+0000", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0043000000000001e-05 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881406+0000", 2024-09-11T05:51:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1670000000000001e-05 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881582+0000", 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017577200000000001 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881649+0000", 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6608999999999999e-05 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884821+0000", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003172336 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884941+0000", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120109 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6616 3.1e e33/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 33'22, mlcod=33'20)", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.404148547999998, 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032823980000000002, 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219136+0000", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.296e-06 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219148+0000", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1257000000000001e-05 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219171+0000", 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3056000000000001e-05 2024-09-11T05:51:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219186+0000", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5684000000000001e-05 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220037+0000", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085078300000000003 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220093+0000", 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5526000000000003e-05 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222304+0000", 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022115070000000001 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222418+0000", 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011328900000000001 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7072 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'24, mlcod=34'22)", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.69219914, 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033394750000000002, 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931087+0000", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931105+0000", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8652999999999999e-05 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931148+0000", 2024-09-11T05:51:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2355999999999999e-05 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931167+0000", 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.929e-05 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931387+0000", 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000220225 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931441+0000", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3808999999999999e-05 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934348+0000", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029071850000000001 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934424+0000", 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5887e-05 2024-09-11T05:51:46.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.630022256, 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7161 3.1e e34/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 34'27, mlcod=34'25)", 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.550848760000001, 2024-09-11T05:51:46.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044832179999999998, 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:46.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072438+0000", 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1599999999999998e-06 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072465+0000", 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6855999999999998e-05 2024-09-11T05:51:46.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072505+0000", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0323e-05 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072527+0000", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2016000000000001e-05 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072689+0000", 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161943 2024-09-11T05:51:46.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072751+0000", 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1241999999999994e-05 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076795+0000", 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040443759999999997 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076918+0000", 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012330199999999999 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:11928 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'62, mlcod=36'60)", 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.018438400000001, 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037441639999999999, 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.651 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604846+0000", 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1230000000000001e-06 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604861+0000", 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4877e-05 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604892+0000", 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0973000000000003e-05 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604915+0000", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2796999999999999e-05 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605079+0000", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016418800000000001 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605138+0000", 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8579999999999998e-05 2024-09-11T05:51:46.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608482+0000", 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033439770000000001 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608590+0000", 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107649 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.655 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:12634 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'87, mlcod=37'85)", 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.102977472999999, 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035953420000000001, 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:46.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520308+0000", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8339999999999999e-06 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520321+0000", 2024-09-11T05:51:46.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2523e-05 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520351+0000", 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0580000000000002e-05 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520366+0000", 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4866e-05 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520499+0000", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132636 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520540+0000", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0883000000000001e-05 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523805+0000", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032651749999999999 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523902+0000", 2024-09-11T05:51:46.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6844999999999999e-05 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15016 3.10 e41/28 3:0f623401:::rbd_data.11243bbd48bb.0000000000000009:head v 41'52, mlcod=39'49)", 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.807591027000001, 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028038719999999998, 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815694+0000", 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1200000000000001e-06 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815958+0000", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026459200000000002 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815992+0000", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3983000000000003e-05 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816009+0000", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.645e-05 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816125+0000", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116152 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816166+0000", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0803000000000002e-05 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818356+0000", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021900629999999999 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818497+0000", 2024-09-11T05:51:46.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014070900000000001 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15346 3.a e41/28 3:5282b638:::rbd_data.11243bbd48bb.0000000000000036:head v 41'128, mlcod=41'126)", 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.350657562999999, 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027676279999999998, 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272629+0000", 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729999999999999e-06 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272646+0000", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7022999999999999e-05 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272688+0000", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2466999999999998e-05 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272709+0000", 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0543e-05 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272878+0000", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169755 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272931+0000", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2889999999999997e-05 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275302+0000", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023706650000000001 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275394+0000", 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1811999999999994e-05 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:16135 3.10 e42/28 3:0a21edc3:::rbd_data.1124d6faf06d.00000000000000d8:head v 42'60, mlcod=42'57)", 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.406746777999999, 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002949171, 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216538+0000", 2024-09-11T05:51:46.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216582+0000", 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3631999999999998e-05 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216607+0000", 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5457e-05 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216624+0000", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6629999999999998e-05 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217303+0000", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000678675 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217354+0000", 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1672999999999998e-05 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219332+0000", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019773249999999998 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219486+0000", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015447200000000001 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:18462 3.1e e45/23 3:7ae7f082:::rbd_data.112453064ca6.0000000000000054:head v 45'53, mlcod=44'50)", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:46.672 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.164679951, 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00273369, 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458604+0000", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458606+0000", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2199999999999999e-06 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458699+0000", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2737000000000003e-05 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458728+0000", 2024-09-11T05:51:46.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9657000000000001e-05 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458743+0000", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4525000000000001e-05 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458902+0000", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015946600000000001 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.458961+0000", 2024-09-11T05:51:46.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9027000000000003e-05 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.461225+0000", 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022638469999999998 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:30.461337+0000", 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112211 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.240305770999999, 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:51:46.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.680 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19804 3.1e e47/23 3:7c44e26f:::rbd_data.11244d01bf21.0000000000000029:head v 47'64, mlcod=47'62)", 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.664415235, 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034400440000000002, 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:46.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958870+0000", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1489999999999999e-06 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958879+0000", 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5599999999999999e-06 2024-09-11T05:51:46.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958898+0000", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9106999999999998e-05 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958909+0000", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0312999999999999e-05 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958983+0000", 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3957999999999999e-05 2024-09-11T05:51:46.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.959009+0000", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6647000000000001e-05 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962253+0000", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032435860000000001 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962309+0000", 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5723999999999997e-05 2024-09-11T05:51:46.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.4194540890000003, 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:51:46.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63804 3.10 e52/28 3:0eb5c07f:::rbd_data.1124767f15a5.0000000000000045:head v 52'107, mlcod=52'105)", 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.3459950169999999, 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032726600000000002, 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277290+0000", 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277670+0000", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037968099999999999 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277690+0000", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0445000000000002e-05 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277700+0000", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6080000000000002e-06 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277756+0000", 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6573999999999997e-05 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277774+0000", 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7258e-05 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280516+0000", 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027427910000000001 2024-09-11T05:51:46.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280561+0000", 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5033000000000001e-05 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63816 3.a e52/28 3:50e92704:::rbd_data.1124767f15a5.0000000000000047:head v 52'166, mlcod=52'164)", 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.3148304780000002, 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033269630000000001, 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.694 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308455+0000", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308910+0000", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045520700000000001 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308930+0000", 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9721e-05 2024-09-11T05:51:46.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308939+0000", 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4369999999999996e-06 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309017+0000", 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7663000000000005e-05 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309041+0000", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4383999999999999e-05 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311746+0000", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027047519999999999 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311780+0000", 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4368e-05 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.698 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.1202750310000003, 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77924 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'171, mlcod=52'169)", 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0142340580000004, 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029501039999999998, 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.703 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609051+0000", 2024-09-11T05:51:46.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609074+0000", 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3286999999999999e-05 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609086+0000", 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2622999999999999e-05 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609093+0000", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609180+0000", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7219000000000006e-05 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609220+0000", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9879000000000002e-05 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.611933+0000", 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00271382 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.612000+0000", 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6334999999999994e-05 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:78189 3.a e55/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 55'179, mlcod=55'177)", 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.865934191, 2024-09-11T05:51:46.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029080249999999998, 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:46.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757352+0000", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2069999999999998e-06 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757368+0000", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5795e-05 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757397+0000", 2024-09-11T05:51:46.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9624999999999998e-05 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757416+0000", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8362000000000001e-05 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757631+0000", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215274 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757685+0000", 2024-09-11T05:51:46.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4203000000000003e-05 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.760140+0000", 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024550380000000001 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.760258+0000", 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117521 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:46.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:46.712 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:46.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:46.713 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:46.712+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:46.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:46.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:46.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:46.807 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T05:51:46.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T05:51:46.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:51:46.844 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:46.841+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:46.891 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:46.892 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:46.892 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:46.892 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:46.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:46.914 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:46.909+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:46.959 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone16 from 0xe69f4b8 (overlap 0x842fcc) to 0x4217e6 2024-09-11T05:51:46.960 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:46.960 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:46.983 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:46.983 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:47.001 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:47.001 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:47.015 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:47.009+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:47.021 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:47.021 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:47.041 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:47.041 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:47.041 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:47.041 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:47.041 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:47.042 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:47.042 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:47.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:47.061 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:47.061 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:47.078 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-09-11T05:51:47.081 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:47.081 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:47.116 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:47.112+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:47.117 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:47.117 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:47.216 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:47.213+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 58, 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout: "created": "2024-09-11T05:50:35.802404+0000", 2024-09-11T05:51:47.229 INFO:teuthology.orchestra.run.smithi165.stdout: "modified": "2024-09-11T05:51:46.831081+0000", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "last_up_change": "2024-09-11T05:50:49.084592+0000", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "last_in_change": "2024-09-11T05:50:38.206686+0000", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_num": 5806080, 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_set": [ 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "nodeep-scrub", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "noscrub", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "pglog_hardlimit", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "purged_snapdirs", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_deletes", 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: "sortbitwise" 2024-09-11T05:51:47.230 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_version": 5, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_snapshot": "", 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_max": 3, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "max_osd": 8, 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "require_min_compat_client": "luminous", 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "min_compat_client": "jewel", 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "squid", 2024-09-11T05:51:47.231 INFO:teuthology.orchestra.run.smithi165.stdout: "allow_crimson": false, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "pools": [ 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 1, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": ".mgr", 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:50.566112+0000", 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 1, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool", 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:47.232 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 1, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 1, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 1, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 1, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 1, 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:47.233 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "18", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:47.234 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:47.235 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:47.236 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "options": { 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_max": 32, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_min": 1 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": {} 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:47.237 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 6.9000000953674316, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 6.9000000953674316, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.28999999165534973, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.238 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 2, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "rbd", 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:54.170388+0000", 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:47.239 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 8, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 8, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 8, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 8, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 8, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:47.240 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "22", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 2, 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 22, 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:47.241 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:47.242 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:47.243 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.630000114440918, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.630000114440918, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1.1399999856948853, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 3, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 3, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:47.244 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "pool_client.0", 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:51:04.331649+0000", 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:51:47.245 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 32, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 32, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 32, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 32, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 32, 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:51:47.246 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "58", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 19, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 58, 2024-09-11T05:51:47.247 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:51:47.248 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:51:47.249 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:51:47.250 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 1.5399999618530273, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 1.5399999618530273, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1.1399999856948853, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 1.75, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 1.75, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: "osds": [ 2024-09-11T05:51:47.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fdc5357-7501-4582-b301-f5eede8a7bc0", 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 56, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.252 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6815", 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6817", 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6818", 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6819", 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.254 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6822", 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6823", 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.255 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6820", 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6821", 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6817/4049823921", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6819/4049823921", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6823/4049823921", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6821/4049823921", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "78f30980-40a1-46fd-bc1a-a656b32210a9", 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.257 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 0.290863037109375, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6800", 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6801", 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6802", 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6803", 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6806", 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.260 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6807", 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6804", 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6805", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6801/2461301418", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6803/2461301418", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6807/2461301418", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6805/2461301418", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fab9ebc-408e-4bc1-b3ad-385d0e396f6b", 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 56, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6808", 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6809", 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6810", 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6811", 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6814", 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6816", 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.266 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6812", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6813", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6809/3166670912", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6811/3166670912", 2024-09-11T05:51:47.267 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6816/3166670912", 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6813/3166670912", 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "d3115ede-8d17-4a27-8629-40b19e3c7ee6", 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.268 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 56, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6824", 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.269 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6825", 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6826", 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.270 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6827", 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6830", 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.271 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6831", 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6828", 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6829", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6825/3322140846", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6827/3322140846", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6831/3322140846", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6829/3322140846", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.273 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "7fa88502-3ffd-4aac-924b-0d6c65737ac6", 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.274 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 53, 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6800", 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6801", 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.275 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6802", 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6803", 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.276 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6806", 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6807", 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6804", 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6805", 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6801/347599006", 2024-09-11T05:51:47.278 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6803/347599006", 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6807/347599006", 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6805/347599006", 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "03238cca-4d2b-412d-9fcd-f489cb4a8455", 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.279 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6816", 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6817", 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6818", 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6819", 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6822", 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.282 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6823", 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6820", 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6821", 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:51:47.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6817/4259095655", 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6819/4259095655", 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6823/4259095655", 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6821/4259095655", 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:51:47.284 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "6c46c2d3-7dd1-4b3a-ad76-391c6e624b40", 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.285 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6824", 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6825", 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.286 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6826", 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6827", 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6830", 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6831", 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6828", 2024-09-11T05:51:47.288 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6829", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6825/2366978310", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6827/2366978310", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6831/2366978310", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6829/2366978310", 2024-09-11T05:51:47.289 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "f849014a-7b82-4361-8a23-3b7a0be5f5c0", 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:51:47.290 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 53, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6808", 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6809", 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6810", 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6811", 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6814", 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6815", 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6812", 2024-09-11T05:51:47.294 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6813", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6809/2576837456", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6811/2576837456", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6815/2576837456", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6813/2576837456", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:51:47.295 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_xinfo": [ 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.605278+0000", 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.681171+0000", 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.297 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.531762+0000", 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:46.158901+0000", 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.412535+0000", 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.299 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.537269+0000", 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.433446+0000", 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.612887+0000", 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap": [], 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_items": [], 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_primaries": [], 2024-09-11T05:51:47.301 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_temp": [], 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_temp": [], 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklist": {}, 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "range_blocklist": {}, 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profiles": { 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "default": { 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "crush-failure-domain": "osd", 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "k": "2", 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "m": "1", 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin": "jerasure", 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "technique": "reed_sol_van" 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps_queue": [], 2024-09-11T05:51:47.302 INFO:teuthology.orchestra.run.smithi165.stdout: "new_removed_snaps": [], 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "new_purged_snaps": [], 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_node_flags": {}, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "device_class_flags": {}, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode": { 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_enabled": false, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_bucket_count": 0, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "degraded_stretch_mode": 0, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "recovering_stretch_mode": 0, 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_bucket": 0 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:51:47.303 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:47.305 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:47.305 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:47.306 INFO:tasks.thrashosds.thrasher:j is {'epoch': 58, 'fsid': 'd5628057-c743-43f5-a51d-1915a16ccfdd', 'created': '2024-09-11T05:50:35.802404+0000', 'modified': '2024-09-11T05:51:46.831081+0000', 'last_up_change': '2024-09-11T05:50:49.084592+0000', 'last_in_change': '2024-09-11T05:50:38.206686+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:50:50.566112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:50:54.170388+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 22, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-11T05:51:04.331649+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2fdc5357-7501-4582-b301-f5eede8a7bc0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6815', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 4049823921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 4049823921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 4049823921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 4049823921}]}, 'public_addr': '172.21.15.165:6817/4049823921', 'cluster_addr': '172.21.15.165:6819/4049823921', 'heartbeat_back_addr': '172.21.15.165:6823/4049823921', 'heartbeat_front_addr': '172.21.15.165:6821/4049823921', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '78f30980-40a1-46fd-bc1a-a656b32210a9', 'up': 1, 'in': 1, 'weight': 0.290863037109375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6800', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2461301418}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2461301418}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 2461301418}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 2461301418}]}, 'public_addr': '172.21.15.165:6801/2461301418', 'cluster_addr': '172.21.15.165:6803/2461301418', 'heartbeat_back_addr': '172.21.15.165:6807/2461301418', 'heartbeat_front_addr': '172.21.15.165:6805/2461301418', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3166670912}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3166670912}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6816', 'nonce': 3166670912}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 3166670912}]}, 'public_addr': '172.21.15.165:6809/3166670912', 'cluster_addr': '172.21.15.165:6811/3166670912', 'heartbeat_back_addr': '172.21.15.165:6816/3166670912', 'heartbeat_front_addr': '172.21.15.165:6813/3166670912', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd3115ede-8d17-4a27-8629-40b19e3c7ee6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 3322140846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 3322140846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 3322140846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 3322140846}]}, 'public_addr': '172.21.15.165:6825/3322140846', 'cluster_addr': '172.21.15.165:6827/3322140846', 'heartbeat_back_addr': '172.21.15.165:6831/3322140846', 'heartbeat_front_addr': '172.21.15.165:6829/3322140846', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 347599006}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 347599006}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 347599006}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 347599006}]}, 'public_addr': '172.21.15.172:6801/347599006', 'cluster_addr': '172.21.15.172:6803/347599006', 'heartbeat_back_addr': '172.21.15.172:6807/347599006', 'heartbeat_front_addr': '172.21.15.172:6805/347599006', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03238cca-4d2b-412d-9fcd-f489cb4a8455', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 4259095655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 4259095655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 4259095655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 4259095655}]}, 'public_addr': '172.21.15.172:6817/4259095655', 'cluster_addr': '172.21.15.172:6819/4259095655', 'heartbeat_back_addr': '172.21.15.172:6823/4259095655', 'heartbeat_front_addr': '172.21.15.172:6821/4259095655', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 2366978310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 2366978310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 2366978310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 2366978310}]}, 'public_addr': '172.21.15.172:6825/2366978310', 'cluster_addr': '172.21.15.172:6827/2366978310', 'heartbeat_back_addr': '172.21.15.172:6831/2366978310', 'heartbeat_front_addr': '172.21.15.172:6829/2366978310', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f849014a-7b82-4361-8a23-3b7a0be5f5c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 2576837456}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2576837456}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2576837456}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 2576837456}]}, 'public_addr': '172.21.15.172:6809/2576837456', 'cluster_addr': '172.21.15.172:6811/2576837456', 'heartbeat_back_addr': '172.21.15.172:6815/2576837456', 'heartbeat_front_addr': '172.21.15.172:6813/2576837456', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.605278+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.681171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.531762+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:46.158901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.412535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.537269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.433446+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.612887+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T05:51:47.306 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-11T05:51:47.306 INFO:teuthology.orchestra.run.smithi165.stdout:2922 trunc from 0xe69f4b8 to 0x3f9ba34 2024-09-11T05:51:47.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:47.346 INFO:teuthology.orchestra.run.smithi165.stdout:2923 trunc from 0x3f9ba34 to 0xd7728fc 2024-09-11T05:51:47.346 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:47.341+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:47.354 INFO:teuthology.orchestra.run.smithi165.stdout:2924 write 0x20f0d6d thru 0x20fadfa (0xa08e bytes) 2024-09-11T05:51:47.360 INFO:teuthology.orchestra.run.smithi165.stdout:2925 punch from 0x4b1e598 to 0x4b2dd5a, (0xf7c2 bytes) 2024-09-11T05:51:47.360 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.360 INFO:teuthology.orchestra.run.smithi165.stdout:2927 write 0x81d8ab2 thru 0x81e1409 (0x8958 bytes) 2024-09-11T05:51:47.365 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.365 INFO:teuthology.orchestra.run.smithi165.stdout:2931 write 0x2fd63eb thru 0x2fe4f7f (0xeb95 bytes) 2024-09-11T05:51:47.372 INFO:teuthology.orchestra.run.smithi165.stdout:2934 writesame 0x5972a22 thru 0x5973986 data_size 0x1(0xf65 bytes) 2024-09-11T05:51:47.379 INFO:teuthology.orchestra.run.smithi165.stdout:2935 write 0x108e9a7 thru 0x1094966 (0x5fc0 bytes) 2024-09-11T05:51:47.390 INFO:teuthology.orchestra.run.smithi165.stdout:2936 trunc from 0xd7728fc to 0xa70d35f 2024-09-11T05:51:47.402 INFO:teuthology.orchestra.run.smithi165.stdout:2937 write 0x8a93582 thru 0x8a93ea8 (0x927 bytes) 2024-09-11T05:51:47.407 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.407 INFO:teuthology.orchestra.run.smithi165.stdout:2939 writesame 0xff4eee thru 0xfffcb8 data_size 0x1(0xadcb bytes) 2024-09-11T05:51:47.418 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:47.416+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:47.470 INFO:teuthology.orchestra.run.smithi165.stdout:2940 writesame 0x82e604a thru 0x82f2a53 data_size 0xb(0xca0a bytes) 2024-09-11T05:51:47.471 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:47.471 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:47.485 INFO:teuthology.orchestra.run.smithi165.stdout:2941 punch from 0x4ab875 to 0x4b447b, (0x8c06 bytes) 2024-09-11T05:51:47.485 INFO:teuthology.orchestra.run.smithi165.stdout:2942 read 0xa596cc4 thru 0xa5a1d01 (0xb03e bytes) 2024-09-11T05:51:47.485 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.485 INFO:teuthology.orchestra.run.smithi165.stdout:2947 read 0x2c703c0 thru 0x2c7c2c9 (0xbf0a bytes) 2024-09-11T05:51:47.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:47.487 INFO:teuthology.orchestra.run.smithi165.stdout:2949 read 0x38f0a11 thru 0x38f8188 (0x7778 bytes) 2024-09-11T05:51:47.487 INFO:teuthology.orchestra.run.smithi165.stdout:2950 read 0x674b5ea thru 0x674c021 (0xa38 bytes) 2024-09-11T05:51:47.487 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.488 INFO:teuthology.orchestra.run.smithi165.stdout:2953 read 0x27a5407 thru 0x27b2d6a (0xd964 bytes) 2024-09-11T05:51:47.499 INFO:teuthology.orchestra.run.smithi165.stdout:2954 write 0xa814e07 thru 0xa81e6cd (0x98c7 bytes) 2024-09-11T05:51:47.505 INFO:teuthology.orchestra.run.smithi165.stdout:2955 punch from 0xa664955 to 0xa66d065, (0x8710 bytes) 2024-09-11T05:51:47.505 INFO:teuthology.orchestra.run.smithi165.stdout:2956 write 0xda9915 thru 0xdadd2e (0x441a bytes) 2024-09-11T05:51:47.510 INFO:teuthology.orchestra.run.smithi165.stdout:2957 write 0x5d65c7a thru 0x5d70aa9 (0xae30 bytes) 2024-09-11T05:51:47.518 INFO:teuthology.orchestra.run.smithi165.stdout:2958 read 0x8f70648 thru 0x8f7a92d (0xa2e6 bytes) 2024-09-11T05:51:47.518 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:47.516+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:47.529 INFO:teuthology.orchestra.run.smithi165.stdout:2959 write 0xb716645 thru 0xb724629 (0xdfe5 bytes) 2024-09-11T05:51:47.535 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.544 INFO:teuthology.orchestra.run.smithi165.stdout:2963 trunc from 0xb72462a to 0x66f333f 2024-09-11T05:51:47.574 INFO:teuthology.orchestra.run.smithi165.stdout:2964 write 0x74db337 thru 0x74dffd1 (0x4c9b bytes) 2024-09-11T05:51:47.580 INFO:teuthology.orchestra.run.smithi165.stdout:2965 read 0x1009e30 thru 0x1013cf9 (0x9eca bytes) 2024-09-11T05:51:47.580 INFO:teuthology.orchestra.run.smithi165.stdout:2966 writesame 0x1f6126d thru 0x1f6ffe1 data_size 0x3(0xed75 bytes) 2024-09-11T05:51:47.611 INFO:teuthology.orchestra.run.smithi165.stdout:2967 punch from 0x4792829 to 0x479f27d, (0xca54 bytes) 2024-09-11T05:51:47.611 INFO:teuthology.orchestra.run.smithi165.stdout:2968 read 0xea204d thru 0xead4cc (0xb480 bytes) 2024-09-11T05:51:47.612 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.612 INFO:teuthology.orchestra.run.smithi165.stdout:2971 writesame 0x58bef2 thru 0x58f46a data_size 0x27(0x3579 bytes) 2024-09-11T05:51:47.620 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:47.616+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:47.621 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:47.621 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:47.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:47.639 INFO:teuthology.orchestra.run.smithi165.stdout:2972 write 0x965e213 thru 0x966ca50 (0xe83e bytes) 2024-09-11T05:51:47.648 INFO:teuthology.orchestra.run.smithi165.stdout:2973 punch from 0x653f6ad to 0x6542e84, (0x37d7 bytes) 2024-09-11T05:51:47.648 INFO:teuthology.orchestra.run.smithi165.stdout:2974 read 0x1c5ca thru 0x1e11d (0x1b54 bytes) 2024-09-11T05:51:47.649 INFO:teuthology.orchestra.run.smithi165.stdout:2975 read 0x95a0812 thru 0x95a7476 (0x6c65 bytes) 2024-09-11T05:51:47.649 INFO:teuthology.orchestra.run.smithi165.stdout:2976 punch from 0xfa5e04 to 0xfaa2e7, (0x44e3 bytes) 2024-09-11T05:51:47.650 INFO:teuthology.orchestra.run.smithi165.stdout:2978 trunc from 0x966ca51 to 0x96be54c 2024-09-11T05:51:47.660 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.681 INFO:teuthology.orchestra.run.smithi165.stdout:2980 writesame 0xedf255c thru 0xee00aa9 data_size 0x31(0xe54e bytes) 2024-09-11T05:51:47.681 INFO:teuthology.orchestra.run.smithi165.stdout:2981 read 0xa2f7d59 thru 0xa2f9f94 (0x223c bytes) 2024-09-11T05:51:47.681 INFO:teuthology.orchestra.run.smithi165.stdout:2982 writesame 0xdb3b40a thru 0xdb3dcf9 data_size 0x28(0x28f0 bytes) 2024-09-11T05:51:47.688 INFO:teuthology.orchestra.run.smithi165.stdout:2983 read 0x19a13fa thru 0x19af73a (0xe341 bytes) 2024-09-11T05:51:47.689 INFO:teuthology.orchestra.run.smithi165.stdout:2984 punch from 0x54540ec to 0x545d927, (0x983b bytes) 2024-09-11T05:51:47.689 INFO:teuthology.orchestra.run.smithi165.stdout:2985 write 0xb52f483 thru 0xb530efa (0x1a78 bytes) 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:2986 punch from 0x8839ea1 to 0x88404fc, (0x665b bytes) 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:2987 punch from 0x38e9456 to 0x38f7642, (0xe1ec bytes) 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:2990 read 0xa9e15c4 thru 0xa9eccfc (0xb739 bytes) 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:2991 punch from 0x81055ab to 0x81133fd, (0xde52 bytes) 2024-09-11T05:51:47.698 INFO:teuthology.orchestra.run.smithi165.stdout:2992 punch from 0xe09c207 to 0xe0a1a98, (0x5891 bytes) 2024-09-11T05:51:47.699 INFO:teuthology.orchestra.run.smithi165.stdout:2993 punch from 0x199a890 to 0x19a1229, (0x6999 bytes) 2024-09-11T05:51:47.699 INFO:teuthology.orchestra.run.smithi165.stdout:2994 writesame 0x9e56049 thru 0x9e5ce7f data_size 0x1b(0x6e37 bytes) 2024-09-11T05:51:47.705 INFO:teuthology.orchestra.run.smithi165.stdout:2995 writesame 0x7c8b0dd thru 0x7c98010 data_size 0x4(0xcf34 bytes) 2024-09-11T05:51:47.719 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:47.717+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:47.721 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.721 INFO:teuthology.orchestra.run.smithi165.stdout:2997 read 0x917e998 thru 0x918035d (0x19c6 bytes) 2024-09-11T05:51:47.721 INFO:teuthology.orchestra.run.smithi165.stdout:2998 write 0x37906a5 thru 0x379b3d0 (0xad2c bytes) 2024-09-11T05:51:47.731 INFO:teuthology.orchestra.run.smithi165.stdout:3001 read 0x54c6b21 thru 0x54ce7fa (0x7cda bytes) 2024-09-11T05:51:47.735 INFO:teuthology.orchestra.run.smithi165.stdout:3002 trunc from 0xee00aaa to 0xd00bf15 2024-09-11T05:51:47.756 INFO:teuthology.orchestra.run.smithi165.stdout:3004 read 0x69324bd thru 0x693a9d0 (0x8514 bytes) 2024-09-11T05:51:47.757 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:47.757 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:47.757 INFO:teuthology.orchestra.run.smithi165.stdout:3005 punch from 0xa04624 to 0xa0fd27, (0xb703 bytes) 2024-09-11T05:51:47.766 INFO:teuthology.orchestra.run.smithi165.stdout:3007 trunc from 0xd00bf15 to 0x85512bf 2024-09-11T05:51:47.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:47.789 INFO:teuthology.orchestra.run.smithi165.stdout:3008 read 0x33ccd4a thru 0x33cf7c2 (0x2a79 bytes) 2024-09-11T05:51:47.790 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.790 INFO:teuthology.orchestra.run.smithi165.stdout:3011 punch from 0x5589db5 to 0x558a1de, (0x429 bytes) 2024-09-11T05:51:47.790 INFO:teuthology.orchestra.run.smithi165.stdout:3012 write 0x567c2a5 thru 0x568aaa3 (0xe7ff bytes) 2024-09-11T05:51:47.798 INFO:teuthology.orchestra.run.smithi165.stdout:3013 writesame 0x73a092d thru 0x73a7aba data_size 0x5f(0x718e bytes) 2024-09-11T05:51:47.807 INFO:teuthology.orchestra.run.smithi165.stdout:3015 trunc from 0x85512bf to 0x5a66ad9 2024-09-11T05:51:47.821 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:47.820+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:47.837 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.837 INFO:teuthology.orchestra.run.smithi165.stdout:3017 read 0x595a186 thru 0x595cd1d (0x2b98 bytes) 2024-09-11T05:51:47.837 INFO:teuthology.orchestra.run.smithi165.stdout:3018 writesame 0x4149f49 thru 0x414acca data_size 0x1(0xd82 bytes) 2024-09-11T05:51:47.842 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:51:47.844 INFO:teuthology.orchestra.run.smithi165.stdout:3019 read 0x51a84e2 thru 0x51b2274 (0x9d93 bytes) 2024-09-11T05:51:47.844 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.844 INFO:teuthology.orchestra.run.smithi165.stdout:3021 punch from 0x2c4fc2f to 0x2c5db7d, (0xdf4e bytes) 2024-09-11T05:51:47.844 INFO:teuthology.orchestra.run.smithi165.stdout:3022 writesame 0x2ac6670 thru 0x2ac6fc5 data_size 0x5(0x956 bytes) 2024-09-11T05:51:47.870 INFO:teuthology.orchestra.run.smithi165.stdout:3023 write 0x6c7bea9 thru 0x6c835c5 (0x771d bytes) 2024-09-11T05:51:47.893 INFO:teuthology.orchestra.run.smithi165.stdout:3024 write 0x94e7a93 thru 0x94ee738 (0x6ca6 bytes) 2024-09-11T05:51:47.899 INFO:teuthology.orchestra.run.smithi165.stdout:3025 punch from 0x4c80cfd to 0x4c8e76e, (0xda71 bytes) 2024-09-11T05:51:47.899 INFO:teuthology.orchestra.run.smithi165.stdout:3026 writesame 0x4aa63c3 thru 0x4ab04e0 data_size 0x29(0xa11e bytes) 2024-09-11T05:51:47.915 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:47.915 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:47.921 INFO:teuthology.orchestra.run.smithi165.stdout:3027 write 0xb4654b3 thru 0xb46d1df (0x7d2d bytes) 2024-09-11T05:51:47.922 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:47.920+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:47.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:47.931 INFO:teuthology.orchestra.run.smithi165.stdout:3028 trunc from 0xb46d1e0 to 0x88aade6 2024-09-11T05:51:47.951 INFO:teuthology.orchestra.run.smithi165.stdout:3029 writesame 0x819a953 thru 0x81a81ae data_size 0x4(0xd85c bytes) 2024-09-11T05:51:47.965 INFO:teuthology.orchestra.run.smithi165.stdout:3030 read 0x59a0036 thru 0x59ad6e8 (0xd6b3 bytes) 2024-09-11T05:51:47.965 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:47.965 INFO:teuthology.orchestra.run.smithi165.stdout:3032 writesame 0x55d2dd5 thru 0x55e2730 data_size 0x2(0xf95c bytes) 2024-09-11T05:51:47.982 INFO:teuthology.orchestra.run.smithi165.stdout:3034 writesame 0x4ccc6e7 thru 0x4ccc94f data_size 0x1(0x269 bytes) 2024-09-11T05:51:47.993 INFO:teuthology.orchestra.run.smithi165.stdout:3035 trunc from 0x88aade6 to 0xb6e95f4 2024-09-11T05:51:48.004 INFO:teuthology.orchestra.run.smithi165.stdout:3036 write 0xc42eb5 thru 0xc486bb (0x5807 bytes) 2024-09-11T05:51:48.013 INFO:teuthology.orchestra.run.smithi165.stdout:3038 read 0xa7e0307 thru 0xa7e4771 (0x446b bytes) 2024-09-11T05:51:48.014 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:48.014 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:48.014 INFO:teuthology.orchestra.run.smithi165.stdout:3042 read 0x1f3d74c thru 0x1f49d06 (0xc5bb bytes) 2024-09-11T05:51:48.014 INFO:teuthology.orchestra.run.smithi165.stdout:3044 write 0x7c19792 thru 0x7c25f2a (0xc799 bytes) 2024-09-11T05:51:48.022 INFO:teuthology.orchestra.run.smithi165.stdout:3045 clone 18 order 19 su 16384 sc 2 2024-09-11T05:51:48.022 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:48.020+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:48.057 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.057 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:48.155 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:48.153+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:48.224 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:48.220+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:48.231 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.231 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:48.325 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:48.320+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:48.412 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.412 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.425 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:48.424+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:48.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:48.526 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:48.521+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:48.542 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.542 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:48.643 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:48.641+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:48.715 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.715 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.727 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:48.721+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:48.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:48.827 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:48.821+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:48.901 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:48.901 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:48.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:48.928 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:48.924+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:49.028 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:49.025+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:49.045 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone17 from 0xb6e95f4 (overlap 0x3f9ba34) to 0x65c5d2 2024-09-11T05:51:49.111 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:49.111 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:49.129 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:49.128+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:49.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:49.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:49.230 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:49.228+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:49.301 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:49.301 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:49.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:49.320 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:49.321 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:49.331 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:49.325+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:49.337 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:49.337 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:49.348 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-09-11T05:51:49.357 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:49.357 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:49.359 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.359 INFO:teuthology.orchestra.run.smithi165.stdout:3047 punch from 0x3fd0ceb to 0x3fd8a8a, (0x7d9f bytes) 2024-09-11T05:51:49.365 INFO:teuthology.orchestra.run.smithi165.stdout:3048 punch from 0x14227bd to 0x142ca4f, (0xa292 bytes) 2024-09-11T05:51:49.371 INFO:teuthology.orchestra.run.smithi165.stdout:3050 trunc from 0xb6e95f4 to 0x7ec2e29 2024-09-11T05:51:49.374 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:49.375 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:49.385 INFO:teuthology.orchestra.run.smithi165.stdout:3051 read 0x55f8d84 thru 0x55f8f08 (0x185 bytes) 2024-09-11T05:51:49.385 INFO:teuthology.orchestra.run.smithi165.stdout:3052 read 0x5049e17 thru 0x50542aa (0xa494 bytes) 2024-09-11T05:51:49.385 INFO:teuthology.orchestra.run.smithi165.stdout:3053 write 0x3ba5331 thru 0x3bac2ee (0x6fbe bytes) 2024-09-11T05:51:49.395 INFO:teuthology.orchestra.run.smithi165.stdout:3054 writesame 0x539e51c thru 0x539effc data_size 0x1(0xae1 bytes) 2024-09-11T05:51:49.396 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:49.396 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:49.411 INFO:teuthology.orchestra.run.smithi165.stdout:3055 write 0xc1cfce7 thru 0xc1db046 (0xb360 bytes) 2024-09-11T05:51:49.417 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:49.417 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:49.419 INFO:teuthology.orchestra.run.smithi165.stdout:3056 punch from 0x9ee70b2 to 0x9eea4e2, (0x3430 bytes) 2024-09-11T05:51:49.429 INFO:teuthology.orchestra.run.smithi165.stdout:3057 trunc from 0xc1db047 to 0x535ba24 2024-09-11T05:51:49.432 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:49.429+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:49.437 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:49.438 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:49.455 INFO:teuthology.orchestra.run.smithi165.stdout:3058 punch from 0x146b0a2 to 0x14739ae, (0x890c bytes) 2024-09-11T05:51:49.459 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:49.460 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:49.463 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:49.463 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:49.463 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:49.463 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:49.465 INFO:teuthology.orchestra.run.smithi165.stdout:3059 writesame 0x73f625a thru 0x74057ff data_size 0x8d(0xf5a6 bytes) 2024-09-11T05:51:49.465 INFO:teuthology.orchestra.run.smithi165.stdout:3060 writesame 0x7095a27 thru 0x70967d5 data_size 0x1(0xdaf bytes) 2024-09-11T05:51:49.472 INFO:teuthology.orchestra.run.smithi165.stdout:3062 read 0xad7940 thru 0xadb686 (0x3d47 bytes) 2024-09-11T05:51:49.473 INFO:teuthology.orchestra.run.smithi165.stdout:3064 write 0xc3bb20 thru 0xc40b5a (0x503b bytes) 2024-09-11T05:51:49.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:49.490 INFO:teuthology.orchestra.run.smithi165.stdout:3065 read 0x4c38b68 thru 0x4c42c7f (0xa118 bytes) 2024-09-11T05:51:49.490 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.490 INFO:teuthology.orchestra.run.smithi165.stdout:3067 writesame 0x6620206 thru 0x662e50b data_size 0x1(0xe306 bytes) 2024-09-11T05:51:49.518 INFO:teuthology.orchestra.run.smithi165.stdout:3069 trunc from 0x7405800 to 0xdda54f 2024-09-11T05:51:49.533 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:49.529+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:49.542 INFO:teuthology.orchestra.run.smithi165.stdout:3070 punch from 0x4bddf6 to 0x4cbdaa, (0xdfb4 bytes) 2024-09-11T05:51:49.543 INFO:teuthology.orchestra.run.smithi165.stdout:3071 flatten 2024-09-11T05:51:49.625 INFO:teuthology.orchestra.run.smithi165.stdout:3072 writesame 0xa8ebc1d thru 0xa8f890e data_size 0x25(0xccf2 bytes) 2024-09-11T05:51:49.626 INFO:teuthology.orchestra.run.smithi165.stdout:3073 read 0x1ecabf8 thru 0x1ed02e0 (0x56e9 bytes) 2024-09-11T05:51:49.626 INFO:teuthology.orchestra.run.smithi165.stdout:3074 punch from 0x2f3357 to 0x2f3388, (0x31 bytes) 2024-09-11T05:51:49.626 INFO:teuthology.orchestra.run.smithi165.stdout:3076 read 0x5c21aeb thru 0x5c2a82e (0x8d44 bytes) 2024-09-11T05:51:49.626 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi165.stdout:3078 trunc from 0xa8f890f to 0x911ce97 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:49.628 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:49.634 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:49.632+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:49.634 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.638 INFO:teuthology.orchestra.run.smithi165.stdout:3083 trunc from 0x911ce97 to 0x64cdba7 2024-09-11T05:51:49.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:49.666 INFO:teuthology.orchestra.run.smithi165.stdout:3084 writesame 0xa74b582 thru 0xa74dd94 data_size 0x1(0x2813 bytes) 2024-09-11T05:51:49.678 INFO:teuthology.orchestra.run.smithi165.stdout:3085 trunc from 0xa74dd95 to 0x4307204 2024-09-11T05:51:49.690 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.697 INFO:teuthology.orchestra.run.smithi165.stdout:3088 trunc from 0x4307204 to 0xeb7b4d 2024-09-11T05:51:49.708 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.709 INFO:teuthology.orchestra.run.smithi165.stdout:3090 trunc from 0xeb7b4d to 0x163e787 2024-09-11T05:51:49.719 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.719 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.736 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:49.729+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:49.745 INFO:teuthology.orchestra.run.smithi165.stdout:3094 write 0x78cf71c thru 0x78dc7ed (0xd0d2 bytes) 2024-09-11T05:51:49.781 INFO:teuthology.orchestra.run.smithi165.stdout:3095 write 0xdf04672 thru 0xdf127f3 (0xe182 bytes) 2024-09-11T05:51:49.793 INFO:teuthology.orchestra.run.smithi165.stdout:3096 read 0xa243429 thru 0xa244d47 (0x191f bytes) 2024-09-11T05:51:49.793 INFO:teuthology.orchestra.run.smithi165.stdout:3097 punch from 0x29c6486 to 0x29cfc37, (0x97b1 bytes) 2024-09-11T05:51:49.804 INFO:teuthology.orchestra.run.smithi165.stdout:3101 write 0xea84fd1 thru 0xea855fb (0x62b bytes) 2024-09-11T05:51:49.812 INFO:teuthology.orchestra.run.smithi165.stdout:3102 punch from 0x6a13190 to 0x6a1d9ba, (0xa82a bytes) 2024-09-11T05:51:49.812 INFO:teuthology.orchestra.run.smithi165.stdout:3103 read 0x8c102e thru 0x8cb597 (0xa56a bytes) 2024-09-11T05:51:49.812 INFO:teuthology.orchestra.run.smithi165.stdout:3105 writesame 0xd153aa6 thru 0xd154b4f data_size 0x1b(0x10aa bytes) 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58950 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7359495650000003, 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064766338000000007, 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58950 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079620+0000", 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1880000000000001e-06 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079627+0000", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2740000000000004e-06 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079633+0000", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1770000000000003e-06 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144235+0000", 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064602407000000001 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144301+0000", 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6125999999999997e-05 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144384+0000", 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2696999999999998e-05 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58951 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 966656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7359276999999995, 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064870423999999996, 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58951 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079642+0000", 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2910000000000002e-06 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079649+0000", 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999999e-06 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079653+0000", 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9439999999999996e-06 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144413+0000", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064759540000000004 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144455+0000", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2438000000000003e-05 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144510+0000", 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4573000000000001e-05 2024-09-11T05:51:49.824 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58952 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7358981890000003, 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065131952000000007, 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.825 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58952 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079670+0000", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079672+0000", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.627e-06 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079680+0000", 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6590000000000007e-06 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079685+0000", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3469999999999998e-06 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144541+0000", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064855876000000007 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144725+0000", 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018391499999999999 2024-09-11T05:51:49.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144801+0000", 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5826000000000002e-05 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58953 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7358742960000004, 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065264442000000006, 2024-09-11T05:51:49.829 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58953 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079694+0000", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2799999999999995e-07 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079696+0000", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0789999999999999e-06 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079702+0000", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.455e-06 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079708+0000", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7749999999999998e-06 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144832+0000", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065124390000000004 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144887+0000", 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4669000000000001e-05 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144957+0000", 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0345999999999994e-05 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58954 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:49.833 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7358526570000006, 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066242835, 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58954 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.08e-07 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079723+0000", 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2830000000000001e-06 2024-09-11T05:51:49.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079730+0000", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6059999999999999e-06 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079735+0000", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1930000000000001e-06 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145779+0000", 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066043973000000006 2024-09-11T05:51:49.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145861+0000", 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1984000000000007e-05 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145957+0000", 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6387999999999999e-05 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58959 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2015232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.837 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7358241020000005, 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065383153999999999, 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58959 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079744+0000", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079746+0000", 2024-09-11T05:51:49.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4830000000000002e-06 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079759+0000", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3113e-05 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079766+0000", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.832e-06 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144985+0000", 2024-09-11T05:51:49.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065218999999999999 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145060+0000", 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5186999999999997e-05 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145126+0000", 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6126999999999999e-05 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.841 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58966 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2093056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7357934650000004, 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065511660999999999, 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58966 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999996e-07 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079777+0000", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6740000000000001e-06 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079782+0000", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7559999999999999e-06 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079787+0000", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3580000000000002e-06 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145153+0000", 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065365728999999997 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145219+0000", 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5720000000000001e-05 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145285+0000", 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6927000000000005e-05 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58972 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2109440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7357753080000009, 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065642100999999994, 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58972 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080613+0000", 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00082077599999999995 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080617+0000", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080627+0000", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.004e-05 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080634+0000", 2024-09-11T05:51:49.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7519999999999999e-06 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145311+0000", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064677068000000004 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145367+0000", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6137999999999997e-05 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145434+0000", 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6931e-05 2024-09-11T05:51:49.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58978 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2187264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7349107670000006, 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064940567000000005, 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58978 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080660+0000", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8629999999999999e-06 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080668+0000", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0879999999999995e-06 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080682+0000", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3788000000000001e-05 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145461+0000", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064779310000000007 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145528+0000", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6764e-05 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145597+0000", 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9052000000000002e-05 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58985 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2234368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7348753650000006, 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065076868999999996, 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58985 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080695+0000", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4959999999999999e-06 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080702+0000", 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3529999999999997e-06 2024-09-11T05:51:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080708+0000", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145624+0000", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064915918000000003 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145693+0000", 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8759999999999999e-05 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145769+0000", 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5862e-05 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58991 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2312192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7348518300000002, 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065155451000000003, 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58991 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080716+0000", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2500000000000001e-07 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080718+0000", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080725+0000", 2024-09-11T05:51:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080730+0000", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2989999999999996e-06 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145793+0000", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065062732999999998 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145831+0000", 2024-09-11T05:51:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8136999999999998e-05 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145871+0000", 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9745e-05 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58997 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7348303569999999, 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065296924000000006, 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58997 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080837+0000", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9950000000000004e-05 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080841+0000", 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2610000000000004e-06 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080850+0000", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5639999999999997e-06 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080856+0000", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.669e-06 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145895+0000", 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065038809000000003 2024-09-11T05:51:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145956+0000", 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0461999999999998e-05 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146034+0000", 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8208999999999996e-05 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58998 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2390016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7346999729999997, 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065346670999999995, 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58998 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081070+0000", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020277699999999999 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081075+0000", 2024-09-11T05:51:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.972e-06 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081085+0000", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9459999999999993e-06 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081093+0000", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9060000000000002e-06 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146084+0000", 2024-09-11T05:51:49.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064990669000000001 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146155+0000", 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1252000000000002e-05 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146214+0000", 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9148999999999999e-05 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59006 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2140160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7344599709999997, 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065251679000000007, 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59006 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081305+0000", 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197519 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081309+0000", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081318+0000", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7900000000000005e-06 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081325+0000", 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4950000000000002e-06 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146244+0000", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064918645999999997 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146303+0000", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8668000000000001e-05 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146359+0000", 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6471999999999997e-05 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.874 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59012 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2217984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.734223879, 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065189569000000003, 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59012 2024-09-11T05:51:49.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082116+0000", 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077256999999999998 2024-09-11T05:51:49.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082144+0000", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7863000000000001e-05 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082163+0000", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9378999999999999e-05 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082195+0000", 2024-09-11T05:51:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1782999999999997e-05 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146393+0000", 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064197562999999999 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146460+0000", 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7197000000000001e-05 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146533+0000", 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3213999999999994e-05 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59024 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2342912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7333314879999993, 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064656276999999998, 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59024 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082244+0000", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1999999999999994e-06 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082252+0000", 2024-09-11T05:51:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.435e-06 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082258+0000", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8610000000000003e-06 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146738+0000", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064479936000000002 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146814+0000", 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6728000000000001e-05 2024-09-11T05:51:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146892+0000", 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7645999999999995e-05 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59031 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2420736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7333014739999992, 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064743414999999999, 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59031 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.63e-07 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082269+0000", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2670000000000001e-06 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082275+0000", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2759999999999997e-06 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082280+0000", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9269999999999996e-06 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146928+0000", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064647901999999993 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146975+0000", 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6860000000000002e-05 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147009+0000", 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.472e-05 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59039 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2437120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7332809339999997, 2024-09-11T05:51:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848283000000007, 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59039 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082342+0000", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5473000000000002e-05 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082346+0000", 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8310000000000003e-06 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082354+0000", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6549999999999993e-06 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082361+0000", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147025+0000", 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064663913000000003 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147082+0000", 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6805000000000002e-05 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147135+0000", 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2438999999999997e-05 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59045 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2514944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7331955570000002, 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064917090999999996, 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59045 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083064+0000", 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069213100000000002 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083069+0000", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2789999999999998e-06 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083081+0000", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2251e-05 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083089+0000", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8140000000000001e-06 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147163+0000", 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064073571999999995 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147228+0000", 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5235999999999997e-05 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147289+0000", 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0807999999999998e-05 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59068 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2592768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7322018069999991, 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064977262999999993, 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59068 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.896 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.084910+0000", 2024-09-11T05:51:49.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001544836 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085422+0000", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051133000000000005 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085433+0000", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085441+0000", 2024-09-11T05:51:49.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179999999999998e-06 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148208+0000", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062766695999999997 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148273+0000", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5395999999999995e-05 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148343+0000", 2024-09-11T05:51:49.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9807000000000004e-05 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:49.900 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:49.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:51:49.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:49.903 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:49.900+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3106 write 0x776b2ce thru 0x777aa45 (0xf778 bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3107 write 0x839406b thru 0x8398ead (0x4e43 bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3108 writesame 0xec5f1b1 thru 0xec6adfa data_size 0x4d(0xbc4a bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3109 write 0x7cac110 thru 0x7cb6a92 (0xa983 bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3110 read 0x11423a0 thru 0x114fadc (0xd73d bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3111 write 0xbde7de9 thru 0xbdf37f2 (0xba0a bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3113 read 0xe362934 thru 0xe367a0f (0x50dc bytes) 2024-09-11T05:51:49.904 INFO:teuthology.orchestra.run.smithi165.stdout:3114 read 0xea9a95a thru 0xeaa53b3 (0xaa5a bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3115 write 0x93d680e thru 0x93e14df (0xacd2 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3117 write 0xe1c75e8 thru 0xe1d4383 (0xcd9c bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3120 writesame 0x85902a thru 0x867f6b data_size 0x46(0xef42 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3121 write 0x8875f61 thru 0x88772c2 (0x1362 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3124 read 0x8a00e37 thru 0x8a0bda3 (0xaf6d bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3125 read 0x98e238f thru 0x98f0e11 (0xea83 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3127 write 0xa31a479 thru 0xa323b6a (0x96f2 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3128 punch from 0x1efde05 to 0x1f0c74b, (0xe946 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3129 writesame 0x9dcf2c0 thru 0x9dd7983 data_size 0x4b(0x86c4 bytes) 2024-09-11T05:51:49.905 INFO:teuthology.orchestra.run.smithi165.stdout:3130 write 0x9424454 thru 0x9427786 (0x3333 bytes) 2024-09-11T05:51:49.927 INFO:teuthology.orchestra.run.smithi165.stdout:3131 trunc from 0xec6adfb to 0x7b71942 2024-09-11T05:51:49.953 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:49.953 INFO:teuthology.orchestra.run.smithi165.stdout:3134 read 0x71596cb thru 0x715d5c3 (0x3ef9 bytes) 2024-09-11T05:51:49.953 INFO:teuthology.orchestra.run.smithi165.stdout:3135 write 0x73f52b5 thru 0x73fd0b3 (0x7dff bytes) 2024-09-11T05:51:49.982 INFO:teuthology.orchestra.run.smithi165.stdout:3136 writesame 0xd7f1f33 thru 0xd7f91d3 data_size 0x1(0x72a1 bytes) 2024-09-11T05:51:49.982 INFO:teuthology.orchestra.run.smithi165.stdout:3137 write 0xbcb0ec8 thru 0xbcbc0be (0xb1f7 bytes) 2024-09-11T05:51:50.001 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:50.000+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:50.011 INFO:teuthology.orchestra.run.smithi165.stdout:3138 trunc from 0xd7f91d4 to 0x32db2c0 2024-09-11T05:51:50.045 INFO:teuthology.orchestra.run.smithi165.stdout:3142 writesame 0x57db0ae thru 0x57e28fc data_size 0x1(0x784f bytes) 2024-09-11T05:51:50.052 INFO:teuthology.orchestra.run.smithi165.stdout:3143 write 0x5ba9b00 thru 0x5bb85a0 (0xeaa1 bytes) 2024-09-11T05:51:50.077 INFO:teuthology.orchestra.run.smithi165.stdout:3144 write 0xd913cb8 thru 0xd91c175 (0x84be bytes) 2024-09-11T05:51:50.080 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:50.080 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:50.083 INFO:teuthology.orchestra.run.smithi165.stdout:3145 read 0x304c1c2 thru 0x3054864 (0x86a3 bytes) 2024-09-11T05:51:50.083 INFO:teuthology.orchestra.run.smithi165.stdout:3147 read 0x1a0ee94 thru 0x1a13c90 (0x4dfd bytes) 2024-09-11T05:51:50.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:50.097 INFO:teuthology.orchestra.run.smithi165.stdout:3148 trunc from 0xd91c176 to 0x3d00e17 2024-09-11T05:51:50.102 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:50.097+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:50.114 INFO:teuthology.orchestra.run.smithi165.stdout:3149 read 0x15327b1 thru 0x153e62e (0xbe7e bytes) 2024-09-11T05:51:50.114 INFO:teuthology.orchestra.run.smithi165.stdout:3150 writesame 0xba4f99 thru 0xbb44aa data_size 0x7f(0xf512 bytes) 2024-09-11T05:51:50.123 INFO:teuthology.orchestra.run.smithi165.stdout:3153 write 0x387a970 thru 0x387f51b (0x4bac bytes) 2024-09-11T05:51:50.156 INFO:teuthology.orchestra.run.smithi165.stdout:3154 write 0x117b5ad thru 0x1182f07 (0x795b bytes) 2024-09-11T05:51:50.188 INFO:teuthology.orchestra.run.smithi165.stdout:3156 write 0xb6c1761 thru 0xb6d0383 (0xec23 bytes) 2024-09-11T05:51:50.196 INFO:teuthology.orchestra.run.smithi165.stdout:3157 punch from 0x5e04cee to 0x5e07fc1, (0x32d3 bytes) 2024-09-11T05:51:50.197 INFO:teuthology.orchestra.run.smithi165.stdout:3158 read 0x200d2bc thru 0x20119bb (0x4700 bytes) 2024-09-11T05:51:50.197 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:50.203 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:50.200+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:50.220 INFO:teuthology.orchestra.run.smithi165.stdout:3160 trunc from 0xb6d0384 to 0x1227f8a 2024-09-11T05:51:50.236 INFO:teuthology.orchestra.run.smithi165.stdout:3161 punch from 0xb6b5be to 0xb6d851, (0x2293 bytes) 2024-09-11T05:51:50.268 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:50.268 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:50.269 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:50.269 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:50.269 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:50.269 INFO:teuthology.orchestra.run.smithi165.stdout:3162 write 0xb7ddad7 thru 0xb7dfbd6 (0x2100 bytes) 2024-09-11T05:51:50.275 INFO:teuthology.orchestra.run.smithi165.stdout:3163 read 0x68393d7 thru 0x683f65a (0x6284 bytes) 2024-09-11T05:51:50.275 INFO:teuthology.orchestra.run.smithi165.stdout:3165 writesame 0x12ce6b5 thru 0x12d8a10 data_size 0x66(0xa35c bytes) 2024-09-11T05:51:50.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:50.288 INFO:teuthology.orchestra.run.smithi165.stdout:3166 trunc from 0xb7dfbd7 to 0xdd67c90 2024-09-11T05:51:50.299 INFO:teuthology.orchestra.run.smithi165.stdout:3167 read 0x96fe316 thru 0x97026dc (0x43c7 bytes) 2024-09-11T05:51:50.299 INFO:teuthology.orchestra.run.smithi165.stdout:3168 write 0xdccf70 thru 0xdce739 (0x17ca bytes) 2024-09-11T05:51:50.302 INFO:teuthology.orchestra.run.smithi165.stdout:3169 clone 19 order 18 su 8192 sc 2 2024-09-11T05:51:50.304 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:50.301+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:50.405 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:50.404+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.694926227, 2024-09-11T05:51:50.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:50.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:50.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.651698113999998, 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68534 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8581620829999999, 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030432710000000002, 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68534 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594459+0000", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594460+0000", 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6700000000000003e-07 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594463+0000", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1420000000000002e-06 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594469+0000", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5150000000000003e-06 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594475+0000", 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.003e-06 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624737+0000", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030261957999999999 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624807+0000", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9529999999999993e-05 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624892+0000", 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4795000000000004e-05 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.469 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8581368779999998, 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:51:50.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:51:50.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:51:50.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:51:50.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8579743860000004, 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.474 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8579493150000008, 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:51:50.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.857925238, 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:51:50.482 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:51:50.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8578745550000004, 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8578449550000009, 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.491 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:51:50.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:51:50.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:51:50.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8578193340000002, 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.857793955, 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.499 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:51:50.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:51:50.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.503 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8577664449999993, 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.507 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8577373280000007, 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:51:50.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8577110650000002, 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.512 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8575166060000008, 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8574934840000008, 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:51:50.520 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:50.524 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8574665059999997, 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8574407470000001, 2024-09-11T05:51:50.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8574136550000002, 2024-09-11T05:51:50.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:50.536 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8573898110000009, 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:50.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:50.540 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:50.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:50.542 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:50.541+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:50.642 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:50.640+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:50.712 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:50.712 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:50.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:50.770 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:50.768+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:50.844 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:50.840+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:50.858 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:51:50.890 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:50.890 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:50.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:50.944 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:50.940+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:51.045 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:51.044+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:1026 3.a e30/28 3:50a9d2c1:::rbd_data.112475afb12f.000000000000004c:head v 30'2, mlcod=0'0)", 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.220447944999997, 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043959489999999997, 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851063+0000", 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899999999999999e-06 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851141+0000", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851182+0000", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1207e-05 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851204+0000", 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1812999999999999e-05 2024-09-11T05:51:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851341+0000", 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013705600000000001 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851391+0000", 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9811999999999997e-05 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855349+0000", 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00395825 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855456+0000", 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107582 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.076 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6402 3.a e33/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 33'21, mlcod=33'19)", 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.190242173000001, 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003674802, 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881269+0000", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6539999999999999e-06 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881325+0000", 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609000000000002e-05 2024-09-11T05:51:51.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881385+0000", 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0043000000000001e-05 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881406+0000", 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1670000000000001e-05 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881582+0000", 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017577200000000001 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881649+0000", 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6608999999999999e-05 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884821+0000", 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003172336 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884941+0000", 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120109 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6616 3.1e e33/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 33'22, mlcod=33'20)", 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.852373253000003, 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032823980000000002, 2024-09-11T05:51:51.081 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219136+0000", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.296e-06 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219148+0000", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1257000000000001e-05 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219171+0000", 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3056000000000001e-05 2024-09-11T05:51:51.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219186+0000", 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5684000000000001e-05 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220037+0000", 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085078300000000003 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220093+0000", 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5526000000000003e-05 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222304+0000", 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022115070000000001 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222418+0000", 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011328900000000001 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7072 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'24, mlcod=34'22)", 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.140423845000001, 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033394750000000002, 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931087+0000", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931105+0000", 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8652999999999999e-05 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931148+0000", 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2355999999999999e-05 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931167+0000", 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.929e-05 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931387+0000", 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000220225 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931441+0000", 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3808999999999999e-05 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934348+0000", 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029071850000000001 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934424+0000", 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5887e-05 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.078246960999998, 2024-09-11T05:51:51.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:51:51.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:51:51.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.094 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7161 3.1e e34/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 34'27, mlcod=34'25)", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.999073465000002, 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044832179999999998, 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072438+0000", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1599999999999998e-06 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072465+0000", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6855999999999998e-05 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072505+0000", 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0323e-05 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072527+0000", 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2016000000000001e-05 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072689+0000", 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161943 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072751+0000", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1241999999999994e-05 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076795+0000", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040443759999999997 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076918+0000", 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012330199999999999 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:11928 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'62, mlcod=36'60)", 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.466663104999999, 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037441639999999999, 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604846+0000", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1230000000000001e-06 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604861+0000", 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4877e-05 2024-09-11T05:51:51.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604892+0000", 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0973000000000003e-05 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604915+0000", 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2796999999999999e-05 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605079+0000", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016418800000000001 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605138+0000", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8579999999999998e-05 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608482+0000", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033439770000000001 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608590+0000", 2024-09-11T05:51:51.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107649 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:12634 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'87, mlcod=37'85)", 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.551202178, 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035953420000000001, 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520308+0000", 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8339999999999999e-06 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520321+0000", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2523e-05 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520351+0000", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0580000000000002e-05 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520366+0000", 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4866e-05 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520499+0000", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132636 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520540+0000", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0883000000000001e-05 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523805+0000", 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032651749999999999 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523902+0000", 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6844999999999999e-05 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15016 3.10 e41/28 3:0f623401:::rbd_data.11243bbd48bb.0000000000000009:head v 41'52, mlcod=39'49)", 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.255815731999999, 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028038719999999998, 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.107 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815693+0000", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815694+0000", 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1200000000000001e-06 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815958+0000", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026459200000000002 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.815992+0000", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3983000000000003e-05 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816009+0000", 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.645e-05 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816125+0000", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116152 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.816166+0000", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0803000000000002e-05 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818356+0000", 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021900629999999999 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:25.818497+0000", 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014070900000000001 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:15346 3.a e41/28 3:5282b638:::rbd_data.11243bbd48bb.0000000000000036:head v 41'128, mlcod=41'126)", 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.798882268, 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027676279999999998, 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.111 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272626+0000", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272629+0000", 2024-09-11T05:51:51.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729999999999999e-06 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272646+0000", 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7022999999999999e-05 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272688+0000", 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2466999999999998e-05 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272709+0000", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0543e-05 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272878+0000", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169755 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.272931+0000", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2889999999999997e-05 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275302+0000", 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023706650000000001 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:26.275394+0000", 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1811999999999994e-05 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:16135 3.10 e42/28 3:0a21edc3:::rbd_data.1124d6faf06d.00000000000000d8:head v 42'60, mlcod=42'57)", 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.854971483, 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002949171, 2024-09-11T05:51:51.115 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216537+0000", 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216538+0000", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216582+0000", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3631999999999998e-05 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216607+0000", 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5457e-05 2024-09-11T05:51:51.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.216624+0000", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6629999999999998e-05 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217303+0000", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000678675 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.217354+0000", 2024-09-11T05:51:51.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1672999999999998e-05 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219332+0000", 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019773249999999998 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:27.219486+0000", 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015447200000000001 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.688530476, 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:51.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:51:51.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:51:51.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:51:51.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19804 3.1e e47/23 3:7c44e26f:::rbd_data.11244d01bf21.0000000000000029:head v 47'64, mlcod=47'62)", 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.112639940000001, 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034400440000000002, 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958870+0000", 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1489999999999999e-06 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958879+0000", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5599999999999999e-06 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958898+0000", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9106999999999998e-05 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958909+0000", 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0312999999999999e-05 2024-09-11T05:51:51.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958983+0000", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3957999999999999e-05 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.959009+0000", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6647000000000001e-05 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962253+0000", 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032435860000000001 2024-09-11T05:51:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962309+0000", 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5723999999999997e-05 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.867678794, 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63804 3.10 e52/28 3:0eb5c07f:::rbd_data.1124767f15a5.0000000000000045:head v 52'107, mlcod=52'105)", 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.794219721999999, 2024-09-11T05:51:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032726600000000002, 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277290+0000", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277670+0000", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037968099999999999 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277690+0000", 2024-09-11T05:51:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0445000000000002e-05 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277700+0000", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6080000000000002e-06 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277756+0000", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6573999999999997e-05 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277774+0000", 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7258e-05 2024-09-11T05:51:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280516+0000", 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027427910000000001 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280561+0000", 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5033000000000001e-05 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63816 3.a e52/28 3:50e92704:::rbd_data.1124767f15a5.0000000000000047:head v 52'166, mlcod=52'164)", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.763055183000001, 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033269630000000001, 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308455+0000", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308910+0000", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045520700000000001 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308930+0000", 2024-09-11T05:51:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9721e-05 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308939+0000", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4369999999999996e-06 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309017+0000", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7663000000000005e-05 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309041+0000", 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4383999999999999e-05 2024-09-11T05:51:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311746+0000", 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027047519999999999 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311780+0000", 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4368e-05 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.5684997359999997, 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77924 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'171, mlcod=52'169)", 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4624587630000008, 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029501039999999998, 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609051+0000", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609074+0000", 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3286999999999999e-05 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609086+0000", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2622999999999999e-05 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609093+0000", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609180+0000", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7219000000000006e-05 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609220+0000", 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9879000000000002e-05 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.611933+0000", 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00271382 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.612000+0000", 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6334999999999994e-05 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:78189 3.a e55/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 55'179, mlcod=55'177)", 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.3141588960000004, 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029080249999999998, 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757350+0000", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757352+0000", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2069999999999998e-06 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757368+0000", 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5795e-05 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757397+0000", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9624999999999998e-05 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757416+0000", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8362000000000001e-05 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757631+0000", 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215274 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.757685+0000", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4203000000000003e-05 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.760140+0000", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024550380000000001 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:44.760258+0000", 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117521 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.152 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:85092 3.0 e60/56 3:06107e3f:::rbd_data.1124fc73d6eb.0000000000000003:head v 60'205, mlcod=60'203)", 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.508094295, 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038279239999999999, 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563416+0000", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8589999999999999e-06 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564065+0000", 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064892599999999997 2024-09-11T05:51:51.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564091+0000", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6299e-05 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564103+0000", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1914e-05 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564179+0000", 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5924e-05 2024-09-11T05:51:51.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564202+0000", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2744000000000001e-05 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567191+0000", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029887300000000002 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567242+0000", 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1527999999999999e-05 2024-09-11T05:51:51.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.157 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:51.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:51.158 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:51.156+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:51.259 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:51.256+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:51.270 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-09-11T05:51:51.303 INFO:teuthology.orchestra.run.smithi165.stdout:3173 read 0xdab16a0 thru 0xdab9c31 (0x8592 bytes) 2024-09-11T05:51:51.305 INFO:teuthology.orchestra.run.smithi165.stdout:3174 writesame 0x6de7d33 thru 0x6df6ac4 data_size 0x2f(0xed92 bytes) 2024-09-11T05:51:51.322 INFO:teuthology.orchestra.run.smithi165.stdout:3175 write 0x24635d8 thru 0x2472f5f (0xf988 bytes) 2024-09-11T05:51:51.332 INFO:teuthology.orchestra.run.smithi165.stdout:3176 punch from 0x89a5922 to 0x89afb78, (0xa256 bytes) 2024-09-11T05:51:51.332 INFO:teuthology.orchestra.run.smithi165.stdout:3177 punch from 0x2d125f5 to 0x2d16b67, (0x4572 bytes) 2024-09-11T05:51:51.332 INFO:teuthology.orchestra.run.smithi165.stdout:3178 read 0x47e09d3 thru 0x47e9a95 (0x90c3 bytes) 2024-09-11T05:51:51.332 INFO:teuthology.orchestra.run.smithi165.stdout:3180 write 0xd7651f3 thru 0xd774e3b (0xfc49 bytes) 2024-09-11T05:51:51.340 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:86148 3.f 3:f5e65b4c:::rbd_data.11244627d646.000000000000035c:head [call rbd.sparse_copyup in=24b,set-alloc-hint object_size 262144 write_size 262144,write 209395~31817 in=31817b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T05:51:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:51.339739+0000", 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.001934484, 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020054230000000001, 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86148 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339739+0000", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339739+0000", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339742+0000", 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.576e-06 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339807+0000", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5140000000000003e-05 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339836+0000", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9391e-05 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339851+0000", 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4708999999999999e-05 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.339959+0000", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108231 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340213+0000", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025398499999999997 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340572+0000", 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000359024 2024-09-11T05:51:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:86149 3.11 3:8a01453c:::rbd_data.11244627d646.000000000000035d:head [call rbd.sparse_copyup in=24b,set-alloc-hint object_size 262144 write_size 262144,write 204800~32768 in=32768b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:51.340539+0000", 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0011341890000000001, 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013968120000000001, 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86149 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340539+0000", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340539+0000", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340541+0000", 2024-09-11T05:51:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6470000000000001e-06 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340588+0000", 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7157000000000003e-05 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340609+0000", 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0435e-05 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340620+0000", 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1926000000000001e-05 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.340754+0000", 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013403199999999999 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.341040+0000", 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028516000000000001 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.341354+0000", 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031396800000000003 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-11T05:51:51.349 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:51.350 INFO:teuthology.orchestra.run.smithi165.stdout:3181 write 0xcfd80f4 thru 0xcfd8ffc (0xf09 bytes) 2024-09-11T05:51:51.350 INFO:teuthology.orchestra.run.smithi165.stdout:3183 punch from 0x6b9cd03 to 0x6ba9bf3, (0xcef0 bytes) 2024-09-11T05:51:51.350 INFO:teuthology.orchestra.run.smithi165.stdout:3184 writesame 0xa3b31fb thru 0xa3c0239 data_size 0x1(0xd03f bytes) 2024-09-11T05:51:51.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:51.359 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:51.356+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:51.398 INFO:teuthology.orchestra.run.smithi165.stdout:3186 clone 20 order 25 su 16777216 sc 9 2024-09-11T05:51:51.460 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:51.460+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:51.475 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:51.475 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:51.475 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:51.475 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:51.475 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:51.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:51.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:51.560 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:51.556+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.507064174, 2024-09-11T05:51:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:51:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:51:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:51:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:51:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.265476790999999, 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.264865965, 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:51:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:51.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.264841558000001, 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:51:51.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:51:51.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.264831239999999, 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:51.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:51:51.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262227625, 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262217183000001, 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262205476, 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.643 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262194396, 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:51.651 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262159301000001, 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:51.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:51:51.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:51:51.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.655 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262118794999999, 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:51.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:51:51.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.659 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262084999000001, 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:51.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:51:51.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:51:51.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.262046096000001, 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:51:51.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:51:51.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.667 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.261959595, 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.668 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.261942334, 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.672 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.26163626, 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:51.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:51:51.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:51:51.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.261576486999999, 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.261554844000001, 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:51:51.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:51:51.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.261533641, 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.26048885, 2024-09-11T05:51:51.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:51.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:51.695 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:51.696 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:51.696 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:51.697 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:51.697 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:51.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:51.698 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:51.698 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:51.699 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:51.699 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:51.699 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:51.696+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:51.701 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:51.701 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:51.721 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:51.721 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:51.741 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:51.741 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:51.762 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:51.762 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:51.818 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:51.818 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:51.819 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:51.816+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:51.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:51.898 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:51.896+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:51.996 INFO:teuthology.orchestra.run.smithi165.stdout:flattening image image_client.0-clone19 2024-09-11T05:51:51.999 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:51.996+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:52.008 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.009 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:52.099 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:52.100+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:52.186 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.186 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.200 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:52.196+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:52.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:52.301 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:52.300+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:52.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:52.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:52.307 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:51:52.307 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-11T05:51:52.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-09-11T05:51:52.366 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.366 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:52.392 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-09-11T05:51:52.402 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:52.400+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:52.503 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:52.500+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:52.534 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.534 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:52.604 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:52.600+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:52.705 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:52.704+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:52.709 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.709 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:52.812 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:52.812+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:52.813 INFO:teuthology.orchestra.run.smithi165.stdout:3188 write 0xcd043f1 thru 0xcd0760a (0x321a bytes) 2024-09-11T05:51:52.830 INFO:teuthology.orchestra.run.smithi165.stdout:3190 writesame 0xaaabcc thru 0xab0c24 data_size 0x1(0x6059 bytes) 2024-09-11T05:51:52.871 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-11T05:51:52.876 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:52.877 INFO:teuthology.orchestra.run.smithi165.stdout:3192 writesame 0x65d420 thru 0x66afe3 data_size 0x14(0xdbc4 bytes) 2024-09-11T05:51:52.882 INFO:teuthology.orchestra.run.smithi165.stdout:3194 read 0x24d2b37 thru 0x24e1917 (0xede1 bytes) 2024-09-11T05:51:52.882 INFO:teuthology.orchestra.run.smithi165.stdout:3195 read 0x5961437 thru 0x5967c2f (0x67f9 bytes) 2024-09-11T05:51:52.882 INFO:teuthology.orchestra.run.smithi165.stdout:3196 write 0x1803be9 thru 0x1810574 (0xc98c bytes) 2024-09-11T05:51:52.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:51:52.900 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:52.900 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:52.907 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:52.904+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:52.912 INFO:teuthology.orchestra.run.smithi165.stdout:3197 trunc from 0xdd67c90 to 0x60f6383 2024-09-11T05:51:52.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:52.948 INFO:teuthology.orchestra.run.smithi165.stdout:3199 write 0x7762224 thru 0x7763de3 (0x1bc0 bytes) 2024-09-11T05:51:52.971 INFO:teuthology.orchestra.run.smithi165.stdout:3200 write 0xa8ab1d0 thru 0xa8add21 (0x2b52 bytes) 2024-09-11T05:51:52.975 INFO:teuthology.orchestra.run.smithi165.stdout:3201 read 0x3940751 thru 0x39492b8 (0x8b68 bytes) 2024-09-11T05:51:52.976 INFO:teuthology.orchestra.run.smithi165.stdout:3203 write 0x97275cd thru 0x97315aa (0x9fde bytes) 2024-09-11T05:51:52.979 INFO:teuthology.orchestra.run.smithi165.stdout:3204 read 0x741f53a thru 0x742da9b (0xe562 bytes) 2024-09-11T05:51:52.983 INFO:teuthology.orchestra.run.smithi165.stdout:3205 trunc from 0xa8add22 to 0xd937e60 2024-09-11T05:51:52.996 INFO:teuthology.orchestra.run.smithi165.stdout:3206 read 0x20f1b thru 0x29f29 (0x900f bytes) 2024-09-11T05:51:53.007 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:53.004+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:53.008 INFO:teuthology.orchestra.run.smithi165.stdout:3209 write 0xde8cb7e thru 0xde8f0c2 (0x2545 bytes) 2024-09-11T05:51:53.011 INFO:teuthology.orchestra.run.smithi165.stdout:3210 writesame 0x218bda7 thru 0x218f030 data_size 0x1(0x328a bytes) 2024-09-11T05:51:53.030 INFO:teuthology.orchestra.run.smithi165.stdout:3211 write 0x6fc35bc thru 0x6fcfd45 (0xc78a bytes) 2024-09-11T05:51:53.033 INFO:teuthology.orchestra.run.smithi165.stdout:3214 write 0xa80593f thru 0xa8099f5 (0x40b7 bytes) 2024-09-11T05:51:53.037 INFO:teuthology.orchestra.run.smithi165.stdout:3216 writesame 0x7d2d5d3 thru 0x7d3723b data_size 0x9(0x9c69 bytes) 2024-09-11T05:51:53.040 INFO:teuthology.orchestra.run.smithi165.stdout:3217 writesame 0x945267a thru 0x945f5fd data_size 0x39(0xcf84 bytes) 2024-09-11T05:51:53.056 INFO:teuthology.orchestra.run.smithi165.stdout:3218 writesame 0xe26010e thru 0xe26fd6e data_size 0x1(0xfc61 bytes) 2024-09-11T05:51:53.061 INFO:teuthology.orchestra.run.smithi165.stdout:3219 writesame 0x348ac3a thru 0x3495bec data_size 0x57(0xafb3 bytes) 2024-09-11T05:51:53.066 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.066 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.066 INFO:teuthology.orchestra.run.smithi165.stdout:3220 write 0x5abdf32 thru 0x5ac4b03 (0x6bd2 bytes) 2024-09-11T05:51:53.070 INFO:teuthology.orchestra.run.smithi165.stdout:3221 punch from 0xb02fcb4 to 0xb0350e4, (0x5430 bytes) 2024-09-11T05:51:53.070 INFO:teuthology.orchestra.run.smithi165.stdout:3222 write 0x82c8d18 thru 0x82d4265 (0xb54e bytes) 2024-09-11T05:51:53.076 INFO:teuthology.orchestra.run.smithi165.stdout:3223 writesame 0x629a725 thru 0x62a97d3 data_size 0x1(0xf0af bytes) 2024-09-11T05:51:53.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:53.110 INFO:teuthology.orchestra.run.smithi165.stdout:3225 punch from 0x5986501 to 0x5994b99, (0xe698 bytes) 2024-09-11T05:51:53.110 INFO:teuthology.orchestra.run.smithi165.stdout:3227 read 0x85313dd thru 0x8535549 (0x416d bytes) 2024-09-11T05:51:53.111 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:53.108+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:53.112 INFO:teuthology.orchestra.run.smithi165.stdout:3228 punch from 0x76bbb87 to 0x76c2214, (0x668d bytes) 2024-09-11T05:51:53.112 INFO:teuthology.orchestra.run.smithi165.stdout:3229 writesame 0x862afc1 thru 0x8639d12 data_size 0x2(0xed52 bytes) 2024-09-11T05:51:53.155 INFO:teuthology.orchestra.run.smithi165.stdout:3231 punch from 0x5ac9c41 to 0x5ad8bf1, (0xefb0 bytes) 2024-09-11T05:51:53.155 INFO:teuthology.orchestra.run.smithi165.stdout:3232 read 0xb3b2eab thru 0xb3b435b (0x14b1 bytes) 2024-09-11T05:51:53.156 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:53.156 INFO:teuthology.orchestra.run.smithi165.stdout:3234 write 0x6bacbe thru 0x6c01ac (0x54ef bytes) 2024-09-11T05:51:53.159 INFO:teuthology.orchestra.run.smithi165.stdout:3235 punch from 0xba78fb8 to 0xba881fc, (0xf244 bytes) 2024-09-11T05:51:53.159 INFO:teuthology.orchestra.run.smithi165.stdout:3236 punch from 0x3eccd9b to 0x3edc02d, (0xf292 bytes) 2024-09-11T05:51:53.159 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:53.160 INFO:teuthology.orchestra.run.smithi165.stdout:3238 read 0x95c45a5 thru 0x95c8f9b (0x49f7 bytes) 2024-09-11T05:51:53.160 INFO:teuthology.orchestra.run.smithi165.stdout:3239 read 0x5af9d7f thru 0x5aff287 (0x5509 bytes) 2024-09-11T05:51:53.160 INFO:teuthology.orchestra.run.smithi165.stdout:3240 write 0x7721a63 thru 0x7726a55 (0x4ff3 bytes) 2024-09-11T05:51:53.163 INFO:teuthology.orchestra.run.smithi165.stdout:3242 read 0x714adc thru 0x7243a1 (0xf8c6 bytes) 2024-09-11T05:51:53.163 INFO:teuthology.orchestra.run.smithi165.stdout:3243 write 0x3dbf2ca thru 0x3dc3cee (0x4a25 bytes) 2024-09-11T05:51:53.167 INFO:teuthology.orchestra.run.smithi165.stdout:3244 read 0x7af7a5a thru 0x7afef09 (0x74b0 bytes) 2024-09-11T05:51:53.167 INFO:teuthology.orchestra.run.smithi165.stdout:3245 read 0x572c738 thru 0x573bede (0xf7a7 bytes) 2024-09-11T05:51:53.168 INFO:teuthology.orchestra.run.smithi165.stdout:3246 write 0x7bfd228 thru 0x7c08bce (0xb9a7 bytes) 2024-09-11T05:51:53.170 INFO:teuthology.orchestra.run.smithi165.stdout:3247 clone 21 order 19 su 16384 sc 8 2024-09-11T05:51:53.209 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:53.208+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:53.223 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.223 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:53.310 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:53.308+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:53.394 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.394 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:53.410 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:53.408+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:53.510 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:53.508+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:53.522 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.522 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:53.624 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:53.624+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:53.696 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.696 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:53.753 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:53.752+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:53.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:53.828 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:53.828+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:53.839 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:53.839 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:53.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:53.889 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:51:53.926 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:53.926 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:53.926 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:53.924+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:53.942 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:53.942 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:53.958 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:53.959 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:53.960 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone20 from 0xe26fd6f (overlap 0x60f6383) to 0x0 2024-09-11T05:51:53.976 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:53.976 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:53.995 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:53.995 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:54.015 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:54.015 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:54.015 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:54.012+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:54.030 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-09-11T05:51:54.031 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:54.032 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:54.034 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:54.034 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:54.034 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:54.035 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:54.035 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:54.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:54.050 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:54.050 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:54.116 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:54.116+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:54.217 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:54.216+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:54.219 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:54.219 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:54.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:54.219 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:54.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58950 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138291572, 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064766338000000007, 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58950 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079618+0000", 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6899999999999998e-07 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079620+0000", 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1880000000000001e-06 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079627+0000", 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2740000000000004e-06 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079633+0000", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1770000000000003e-06 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144235+0000", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064602407000000001 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144301+0000", 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6125999999999997e-05 2024-09-11T05:51:54.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144384+0000", 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2696999999999998e-05 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58951 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 966656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138269706999999, 2024-09-11T05:51:54.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064870423999999996, 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58951 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079640+0000", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079642+0000", 2024-09-11T05:51:54.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2910000000000002e-06 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079649+0000", 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999999e-06 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079653+0000", 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9439999999999996e-06 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144413+0000", 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064759540000000004 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144455+0000", 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2438000000000003e-05 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144510+0000", 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4573000000000001e-05 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58952 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138240196, 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065131952000000007, 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.229 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58952 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079669+0000", 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079670+0000", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079672+0000", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.627e-06 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079680+0000", 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6590000000000007e-06 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079685+0000", 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3469999999999998e-06 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144541+0000", 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064855876000000007 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144725+0000", 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018391499999999999 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144801+0000", 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5826000000000002e-05 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58953 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:54.233 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138216303, 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065264442000000006, 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58953 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:54.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079693+0000", 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079694+0000", 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2799999999999995e-07 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079696+0000", 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0789999999999999e-06 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079702+0000", 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.455e-06 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079708+0000", 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7749999999999998e-06 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144832+0000", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065124390000000004 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144887+0000", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4669000000000001e-05 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144957+0000", 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0345999999999994e-05 2024-09-11T05:51:54.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58954 3.a 3:54e008d8:::rbd_data.1124b2f65e38.000000000000001b:11 [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138194664, 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066242835, 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58954 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079715+0000", 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.08e-07 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079723+0000", 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2830000000000001e-06 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079730+0000", 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6059999999999999e-06 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079735+0000", 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1930000000000001e-06 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145779+0000", 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066043973000000006 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145861+0000", 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1984000000000007e-05 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145957+0000", 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6387999999999999e-05 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58959 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2015232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138166109, 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065383153999999999, 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58959 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079743+0000", 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079744+0000", 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079746+0000", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4830000000000002e-06 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079759+0000", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3113e-05 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079766+0000", 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.832e-06 2024-09-11T05:51:54.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.144985+0000", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065218999999999999 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145060+0000", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5186999999999997e-05 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145126+0000", 2024-09-11T05:51:54.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6126999999999999e-05 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58966 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2093056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138135472, 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065511660999999999, 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58966 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:54.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079774+0000", 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999996e-07 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079777+0000", 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6740000000000001e-06 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079782+0000", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7559999999999999e-06 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079787+0000", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3580000000000002e-06 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145153+0000", 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065365728999999997 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145219+0000", 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5720000000000001e-05 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145285+0000", 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6927000000000005e-05 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.251 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58972 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2109440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.138117315000001, 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065642100999999994, 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58972 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.079792+0000", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080613+0000", 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00082077599999999995 2024-09-11T05:51:54.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080617+0000", 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080627+0000", 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.004e-05 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080634+0000", 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7519999999999999e-06 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145311+0000", 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064677068000000004 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145367+0000", 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6137999999999997e-05 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145434+0000", 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6931e-05 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58978 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2187264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.137252774, 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064940567000000005, 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58978 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080657+0000", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0200000000000001e-07 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080660+0000", 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8629999999999999e-06 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080668+0000", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0879999999999995e-06 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080682+0000", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3788000000000001e-05 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145461+0000", 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064779310000000007 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145528+0000", 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6764e-05 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145597+0000", 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9052000000000002e-05 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58985 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2234368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.137217372, 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065076868999999996, 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58985 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080692+0000", 2024-09-11T05:51:54.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080695+0000", 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4959999999999999e-06 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080702+0000", 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3529999999999997e-06 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080708+0000", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145624+0000", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064915918000000003 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145693+0000", 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8759999999999999e-05 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145769+0000", 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5862e-05 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58991 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2312192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.137193837, 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065155451000000003, 2024-09-11T05:51:54.265 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58991 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:54.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080715+0000", 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080716+0000", 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2500000000000001e-07 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080718+0000", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080725+0000", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080730+0000", 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2989999999999996e-06 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145793+0000", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065062732999999998 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145831+0000", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8136999999999998e-05 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145871+0000", 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9745e-05 2024-09-11T05:51:54.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58997 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.137172364, 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065296924000000006, 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58997 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080737+0000", 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080837+0000", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9950000000000004e-05 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080841+0000", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2610000000000004e-06 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080850+0000", 2024-09-11T05:51:54.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5639999999999997e-06 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080856+0000", 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.669e-06 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145895+0000", 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065038809000000003 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.145956+0000", 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0461999999999998e-05 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146034+0000", 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8208999999999996e-05 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:58998 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2390016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.137041979999999, 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065346670999999995, 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58998 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:54.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.080867+0000", 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081070+0000", 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020277699999999999 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081075+0000", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.972e-06 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081085+0000", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9459999999999993e-06 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081093+0000", 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9060000000000002e-06 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146084+0000", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064990669000000001 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146155+0000", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1252000000000002e-05 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146214+0000", 2024-09-11T05:51:54.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9148999999999999e-05 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59006 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2140160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.136801977999999, 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065251679000000007, 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.279 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59006 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081107+0000", 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081305+0000", 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000197519 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081309+0000", 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081318+0000", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7900000000000005e-06 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081325+0000", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4950000000000002e-06 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146244+0000", 2024-09-11T05:51:54.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064918645999999997 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146303+0000", 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8668000000000001e-05 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146359+0000", 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6471999999999997e-05 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59012 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2217984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.136565886, 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065189569000000003, 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59012 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.081343+0000", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082116+0000", 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077256999999999998 2024-09-11T05:51:54.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082144+0000", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7863000000000001e-05 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082163+0000", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9378999999999999e-05 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082195+0000", 2024-09-11T05:51:54.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1782999999999997e-05 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146393+0000", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064197562999999999 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146460+0000", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7197000000000001e-05 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146533+0000", 2024-09-11T05:51:54.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3213999999999994e-05 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59024 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2342912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.135673495000001, 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064656276999999998, 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59024 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082236+0000", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082244+0000", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1999999999999994e-06 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082252+0000", 2024-09-11T05:51:54.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.435e-06 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082258+0000", 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8610000000000003e-06 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146738+0000", 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064479936000000002 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146814+0000", 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6728000000000001e-05 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146892+0000", 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7645999999999995e-05 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59031 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2420736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.135643481000001, 2024-09-11T05:51:54.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064743414999999999, 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59031 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082266+0000", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.63e-07 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082269+0000", 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2670000000000001e-06 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082275+0000", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2759999999999997e-06 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082280+0000", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9269999999999996e-06 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146928+0000", 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064647901999999993 2024-09-11T05:51:54.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.146975+0000", 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6860000000000002e-05 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147009+0000", 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.472e-05 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59039 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2437120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.135622940999999, 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848283000000007, 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59039 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082286+0000", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082342+0000", 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5473000000000002e-05 2024-09-11T05:51:54.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082346+0000", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8310000000000003e-06 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082354+0000", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6549999999999993e-06 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082361+0000", 2024-09-11T05:51:54.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147025+0000", 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064663913000000003 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147082+0000", 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6805000000000002e-05 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147135+0000", 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2438999999999997e-05 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59045 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2514944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.135537564, 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064917090999999996, 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59045 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.082372+0000", 2024-09-11T05:51:54.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083064+0000", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069213100000000002 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083069+0000", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2789999999999998e-06 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083081+0000", 2024-09-11T05:51:54.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2251e-05 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083089+0000", 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8140000000000001e-06 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147163+0000", 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064073571999999995 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147228+0000", 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5235999999999997e-05 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.147289+0000", 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0807999999999998e-05 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:59068 3.1a 3:5ec56583:::rbd_data.1124a9e466a.0000000000000008:10 [read 2592768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.134543814000001, 2024-09-11T05:51:54.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064977262999999993, 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 59068 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.083365+0000", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.084910+0000", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001544836 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085422+0000", 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051133000000000005 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085433+0000", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.085441+0000", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179999999999998e-06 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148208+0000", 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062766695999999997 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148273+0000", 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5395999999999995e-05 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.148343+0000", 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9807000000000004e-05 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.309 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:54.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:54.318 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:54.316+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:54.420 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:54.416+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:54.461 INFO:teuthology.orchestra.run.smithi165.stdout:3248 read 0x34eeebf thru 0x34f4d24 (0x5e66 bytes) 2024-09-11T05:51:54.484 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:54.484 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:54.485 INFO:teuthology.orchestra.run.smithi165.stdout:3249 writesame 0xed1f79e thru 0xed242fb data_size 0xb(0x4b5e bytes) 2024-09-11T05:51:54.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:54.500 INFO:teuthology.orchestra.run.smithi165.stdout:3250 trunc from 0xed242fc to 0x4804cbc 2024-09-11T05:51:54.520 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:54.520+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:54.529 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:54.549 INFO:teuthology.orchestra.run.smithi165.stdout:3252 writesame 0xae5ff19 thru 0xae6741b data_size 0x1(0x7503 bytes) 2024-09-11T05:51:54.550 INFO:teuthology.orchestra.run.smithi165.stdout:3253 write 0x5eccbda thru 0x5ed4cfe (0x8125 bytes) 2024-09-11T05:51:54.560 INFO:teuthology.orchestra.run.smithi165.stdout:3254 read 0x14a1847 thru 0x14a911d (0x78d7 bytes) 2024-09-11T05:51:54.561 INFO:teuthology.orchestra.run.smithi165.stdout:3255 read 0x593521b thru 0x593c357 (0x713d bytes) 2024-09-11T05:51:54.561 INFO:teuthology.orchestra.run.smithi165.stdout:3258 punch from 0x1dd980 to 0x1e1db7, (0x4437 bytes) 2024-09-11T05:51:54.575 INFO:teuthology.orchestra.run.smithi165.stdout:3259 writesame 0xd10030e thru 0xd108fac data_size 0x1(0x8c9f bytes) 2024-09-11T05:51:54.576 INFO:teuthology.orchestra.run.smithi165.stdout:3261 read 0xc6487ac thru 0xc657124 (0xe979 bytes) 2024-09-11T05:51:54.576 INFO:teuthology.orchestra.run.smithi165.stdout:3262 punch from 0x848399e to 0x848da8b, (0xa0ed bytes) 2024-09-11T05:51:54.576 INFO:teuthology.orchestra.run.smithi165.stdout:3263 read 0x26a47f0 thru 0x26afd4c (0xb55d bytes) 2024-09-11T05:51:54.576 INFO:teuthology.orchestra.run.smithi165.stdout:3264 punch from 0xa2bfbb2 to 0xa2c1526, (0x1974 bytes) 2024-09-11T05:51:54.577 INFO:teuthology.orchestra.run.smithi165.stdout:3265 write 0xb0d1aaa thru 0xb0d37e3 (0x1d3a bytes) 2024-09-11T05:51:54.582 INFO:teuthology.orchestra.run.smithi165.stdout:3266 writesame 0x7bd0b4b thru 0x7bd4c96 data_size 0x2a(0x414c bytes) 2024-09-11T05:51:54.592 INFO:teuthology.orchestra.run.smithi165.stdout:3267 read 0x52feb22 thru 0x53080d9 (0x95b8 bytes) 2024-09-11T05:51:54.592 INFO:teuthology.orchestra.run.smithi165.stdout:3268 read 0x97f04e7 thru 0x97f4d01 (0x481b bytes) 2024-09-11T05:51:54.592 INFO:teuthology.orchestra.run.smithi165.stdout:3269 write 0xb36a2a3 thru 0xb377c18 (0xd976 bytes) 2024-09-11T05:51:54.602 INFO:teuthology.orchestra.run.smithi165.stdout:3270 writesame 0x892e24c thru 0x893d16f data_size 0xa(0xef24 bytes) 2024-09-11T05:51:54.617 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:54.617 INFO:teuthology.orchestra.run.smithi165.stdout:3274 write 0xc262cab thru 0xc2729bf (0xfd15 bytes) 2024-09-11T05:51:54.621 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:54.620+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:54.629 INFO:teuthology.orchestra.run.smithi165.stdout:3276 read 0x23c87fc thru 0x23ca1aa (0x19af bytes) 2024-09-11T05:51:54.630 INFO:teuthology.orchestra.run.smithi165.stdout:3277 writesame 0x233f132 thru 0x2346fde data_size 0x1(0x7ead bytes) 2024-09-11T05:51:54.664 INFO:teuthology.orchestra.run.smithi165.stdout:3278 read 0x6aec77 thru 0x6b2790 (0x3b1a bytes) 2024-09-11T05:51:54.679 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:54.679 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:54.679 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:54.679 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:54.679 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:54.684 INFO:teuthology.orchestra.run.smithi165.stdout:3280 trunc from 0xd108fad to 0x43ad183 2024-09-11T05:51:54.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:54.722 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:54.720+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:54.771 INFO:teuthology.orchestra.run.smithi165.stdout:3281 punch from 0x1585f0f to 0x158ef2f, (0x9020 bytes) 2024-09-11T05:51:54.782 INFO:teuthology.orchestra.run.smithi165.stdout:3282 trunc from 0x43ad183 to 0x8ef740 2024-09-11T05:51:54.822 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:54.820+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.075020872000003, 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.031792758999998, 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.238231523, 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:51:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.238069031, 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:51:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.238043960000001, 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.238019883, 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:51:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:51:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.2379692, 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:51:54.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.868 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237939600000001, 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:51:54.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237913979, 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:51:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.2378886, 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:51:54.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237861089999999, 2024-09-11T05:51:54.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.886 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237831973, 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.887 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:51:54.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:51:54.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.23780571, 2024-09-11T05:51:54.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:51:54.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:51:54.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237611251000001, 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:51:54.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237588129000001, 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:51:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:51:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237561151, 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:51:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237535392, 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:51:54.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:51:54.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.2375083, 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:51:54.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:51:54.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237484456000001, 2024-09-11T05:51:54.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:54.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:51:54.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:51:54.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:86155 3.1d 3:bed54d18:::rbd_data.11244627d646.000000000000028f:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.4768323959999998, 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043742621000000002, 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:54.924 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86155 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:54.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355884+0000", 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3340000000000001e-06 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355890+0000", 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7679999999999997e-06 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355903+0000", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.259e-05 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355909+0000", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9349999999999999e-06 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356023+0000", 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114307 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356362+0000", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033904300000000003 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.365362+0000", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089999659999999999 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399542+0000", 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034180413999999999 2024-09-11T05:51:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399557+0000", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4681999999999999e-05 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399594+0000", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.646e-05 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399626+0000", 2024-09-11T05:51:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2122000000000002e-05 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:54.930 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:54.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:54.932 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:54.928+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:54.938 INFO:teuthology.orchestra.run.smithi165.stdout:3283 write 0x2b63f10 thru 0x2b65d3d (0x1e2e bytes) 2024-09-11T05:51:54.960 INFO:teuthology.orchestra.run.smithi165.stdout:3284 trunc from 0x2b65d3e to 0x7a66d69 2024-09-11T05:51:54.995 INFO:teuthology.orchestra.run.smithi165.stdout:3285 writesame 0xe5c72e2 thru 0xe5d4c96 data_size 0x1(0xd9b5 bytes) 2024-09-11T05:51:54.995 INFO:teuthology.orchestra.run.smithi165.stdout:3286 read 0x5c042da thru 0x5c1110e (0xce35 bytes) 2024-09-11T05:51:54.995 INFO:teuthology.orchestra.run.smithi165.stdout:3287 writesame 0x4195d33 thru 0x419e4e9 data_size 0x25(0x87b7 bytes) 2024-09-11T05:51:55.033 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:55.032+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:55.033 INFO:teuthology.orchestra.run.smithi165.stdout:3288 trunc from 0xe5d4c97 to 0x335721e 2024-09-11T05:51:55.071 INFO:teuthology.orchestra.run.smithi165.stdout:3289 writesame 0x718e9e9 thru 0x719177f data_size 0x1(0x2d97 bytes) 2024-09-11T05:51:55.071 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.090 INFO:teuthology.orchestra.run.smithi165.stdout:3291 trunc from 0x7191780 to 0xe938818 2024-09-11T05:51:55.105 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:55.105 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:55.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:55.122 INFO:teuthology.orchestra.run.smithi165.stdout:3292 trunc from 0xe938818 to 0x5e8e4d3 2024-09-11T05:51:55.132 INFO:teuthology.orchestra.run.smithi165.stdout:3293 read 0x11189a9 thru 0x1121ead (0x9505 bytes) 2024-09-11T05:51:55.133 INFO:teuthology.orchestra.run.smithi165.stdout:3294 punch from 0x44b5227 to 0x44ba6d0, (0x54a9 bytes) 2024-09-11T05:51:55.133 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:55.132+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:55.161 INFO:teuthology.orchestra.run.smithi165.stdout:3295 write 0xe8128c6 thru 0xe81b9da (0x9115 bytes) 2024-09-11T05:51:55.169 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.169 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.169 INFO:teuthology.orchestra.run.smithi165.stdout:3300 write 0x77ce7d9 thru 0x77dc8a4 (0xe0cc bytes) 2024-09-11T05:51:55.183 INFO:teuthology.orchestra.run.smithi165.stdout:3301 read 0xddaf291 thru 0xddb5238 (0x5fa8 bytes) 2024-09-11T05:51:55.201 INFO:teuthology.orchestra.run.smithi165.stdout:3302 trunc from 0xe81b9db to 0x783fd6a 2024-09-11T05:51:55.231 INFO:teuthology.orchestra.run.smithi165.stdout:3303 writesame 0x81471e9 thru 0x814df15 data_size 0x1(0x6d2d bytes) 2024-09-11T05:51:55.234 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:55.232+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:55.248 INFO:teuthology.orchestra.run.smithi165.stdout:3304 writesame 0xa024d73 thru 0xa02ab78 data_size 0x53(0x5e06 bytes) 2024-09-11T05:51:55.249 INFO:teuthology.orchestra.run.smithi165.stdout:3305 writesame 0x491c895 thru 0x492c3f7 data_size 0x5(0xfb63 bytes) 2024-09-11T05:51:55.261 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:55.261 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:55.261 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:55.261 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:55.261 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:55.271 INFO:teuthology.orchestra.run.smithi165.stdout:3306 write 0x655bae2 thru 0x6569600 (0xdb1f bytes) 2024-09-11T05:51:55.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:55.286 INFO:teuthology.orchestra.run.smithi165.stdout:3308 read 0x8d7bc19 thru 0x8d853aa (0x9792 bytes) 2024-09-11T05:51:55.286 INFO:teuthology.orchestra.run.smithi165.stdout:3309 read 0x4478e05 thru 0x44871c9 (0xe3c5 bytes) 2024-09-11T05:51:55.286 INFO:teuthology.orchestra.run.smithi165.stdout:3310 read 0x7fac02 thru 0x80623b (0xb63a bytes) 2024-09-11T05:51:55.309 INFO:teuthology.orchestra.run.smithi165.stdout:3311 writesame 0xe232d86 thru 0xe23f33b data_size 0x1(0xc5b6 bytes) 2024-09-11T05:51:55.309 INFO:teuthology.orchestra.run.smithi165.stdout:3312 read 0xe1b7c35 thru 0xe1c5e82 (0xe24e bytes) 2024-09-11T05:51:55.309 INFO:teuthology.orchestra.run.smithi165.stdout:3316 writesame 0x305dc6 thru 0x3098cb data_size 0x5(0x3b06 bytes) 2024-09-11T05:51:55.334 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:55.332+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:55.363 INFO:teuthology.orchestra.run.smithi165.stdout:3317 write 0x3c077eb thru 0x3c0c7a5 (0x4fbb bytes) 2024-09-11T05:51:55.372 INFO:teuthology.orchestra.run.smithi165.stdout:3318 writesame 0x7f76ca4 thru 0x7f86789 data_size 0xf(0xfae6 bytes) 2024-09-11T05:51:55.387 INFO:teuthology.orchestra.run.smithi165.stdout:3319 write 0x2226a1 thru 0x2280bb (0x5a1b bytes) 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:1026 3.a e30/28 3:50a9d2c1:::rbd_data.112475afb12f.000000000000004c:head v 30'2, mlcod=0'0)", 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.554790443999998, 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043959489999999997, 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851063+0000", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899999999999999e-06 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851141+0000", 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851182+0000", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1207e-05 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851204+0000", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1812999999999999e-05 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851341+0000", 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013705600000000001 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851391+0000", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9811999999999997e-05 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855349+0000", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00395825 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855456+0000", 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107582 2024-09-11T05:51:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6402 3.a e33/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 33'21, mlcod=33'19)", 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.524584672000003, 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003674802, 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881269+0000", 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6539999999999999e-06 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881325+0000", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609000000000002e-05 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881385+0000", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0043000000000001e-05 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881406+0000", 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1670000000000001e-05 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881582+0000", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017577200000000001 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881649+0000", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6608999999999999e-05 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884821+0000", 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003172336 2024-09-11T05:51:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884941+0000", 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120109 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6616 3.1e e33/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 33'22, mlcod=33'20)", 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.186715751999998, 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032823980000000002, 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219135+0000", 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219136+0000", 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.296e-06 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219148+0000", 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1257000000000001e-05 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219171+0000", 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3056000000000001e-05 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.219186+0000", 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5684000000000001e-05 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220037+0000", 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085078300000000003 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.220093+0000", 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5526000000000003e-05 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222304+0000", 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022115070000000001 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.222418+0000", 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011328900000000001 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7072 3.10 e34/28 3:09743d5d:::rbd_data.11249b359231.000000000000001b:head v 34'24, mlcod=34'22)", 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.474766344000003, 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033394750000000002, 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931085+0000", 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931087+0000", 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931105+0000", 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8652999999999999e-05 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931148+0000", 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2355999999999999e-05 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931167+0000", 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.929e-05 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931387+0000", 2024-09-11T05:51:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000220225 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.931441+0000", 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3808999999999999e-05 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934348+0000", 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029071850000000001 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.934424+0000", 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5887e-05 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.41258946, 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:51:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:51:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7161 3.1e e34/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 34'27, mlcod=34'25)", 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.333415963999997, 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044832179999999998, 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072438+0000", 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1599999999999998e-06 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072465+0000", 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6855999999999998e-05 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072505+0000", 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0323e-05 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072527+0000", 2024-09-11T05:51:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2016000000000001e-05 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072689+0000", 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161943 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072751+0000", 2024-09-11T05:51:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1241999999999994e-05 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076795+0000", 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040443759999999997 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076918+0000", 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012330199999999999 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:11928 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'62, mlcod=36'60)", 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.801005603999997, 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037441639999999999, 2024-09-11T05:51:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604846+0000", 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1230000000000001e-06 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604861+0000", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4877e-05 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604892+0000", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0973000000000003e-05 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604915+0000", 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2796999999999999e-05 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605079+0000", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016418800000000001 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605138+0000", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8579999999999998e-05 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608482+0000", 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033439770000000001 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608590+0000", 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107649 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:12634 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'87, mlcod=37'85)", 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.885544676999999, 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035953420000000001, 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520308+0000", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8339999999999999e-06 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520321+0000", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2523e-05 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520351+0000", 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0580000000000002e-05 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520366+0000", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4866e-05 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520499+0000", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132636 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520540+0000", 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0883000000000001e-05 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523805+0000", 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032651749999999999 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523902+0000", 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6844999999999999e-05 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.022872974999999, 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:51:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:51:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:51:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19804 3.1e e47/23 3:7c44e26f:::rbd_data.11244d01bf21.0000000000000029:head v 47'64, mlcod=47'62)", 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.446982438999999, 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034400440000000002, 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958869+0000", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958870+0000", 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1489999999999999e-06 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958879+0000", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5599999999999999e-06 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958898+0000", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9106999999999998e-05 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958909+0000", 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0312999999999999e-05 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.958983+0000", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3957999999999999e-05 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.959009+0000", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6647000000000001e-05 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962253+0000", 2024-09-11T05:51:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032435860000000001 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.962309+0000", 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5723999999999997e-05 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.202021293, 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:51:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:51:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63804 3.10 e52/28 3:0eb5c07f:::rbd_data.1124767f15a5.0000000000000045:head v 52'107, mlcod=52'105)", 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.128562220999999, 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032726600000000002, 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.462 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277289+0000", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277290+0000", 2024-09-11T05:51:55.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277670+0000", 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037968099999999999 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277690+0000", 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0445000000000002e-05 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277700+0000", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6080000000000002e-06 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277756+0000", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6573999999999997e-05 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.277774+0000", 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7258e-05 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280516+0000", 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027427910000000001 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.280561+0000", 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5033000000000001e-05 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:63816 3.a e52/28 3:50e92704:::rbd_data.1124767f15a5.0000000000000047:head v 52'166, mlcod=52'164)", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.097397682, 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033269630000000001, 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308453+0000", 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308455+0000", 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308910+0000", 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045520700000000001 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308930+0000", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9721e-05 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.308939+0000", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4369999999999996e-06 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309017+0000", 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7663000000000005e-05 2024-09-11T05:51:55.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.309041+0000", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4383999999999999e-05 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311746+0000", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027047519999999999 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.311780+0000", 2024-09-11T05:51:55.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4368e-05 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.902842235, 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:51:55.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:51:55.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77924 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'171, mlcod=52'169)", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:55.475 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.796801262000001, 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029501039999999998, 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609050+0000", 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609051+0000", 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609074+0000", 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3286999999999999e-05 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609086+0000", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2622999999999999e-05 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609093+0000", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1290000000000001e-06 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609180+0000", 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7219000000000006e-05 2024-09-11T05:51:55.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.609220+0000", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9879000000000002e-05 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.611933+0000", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00271382 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.612000+0000", 2024-09-11T05:51:55.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6334999999999994e-05 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:85092 3.0 e60/56 3:06107e3f:::rbd_data.1124fc73d6eb.0000000000000003:head v 60'205, mlcod=60'203)", 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.8424367940000002, 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038279239999999999, 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563416+0000", 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8589999999999999e-06 2024-09-11T05:51:55.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564065+0000", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064892599999999997 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564091+0000", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6299e-05 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564103+0000", 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1914e-05 2024-09-11T05:51:55.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564179+0000", 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5924e-05 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564202+0000", 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2744000000000001e-05 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567191+0000", 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029887300000000002 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567242+0000", 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1527999999999999e-05 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:89647 3.a e64/28 3:538a4bad:::rbd_data.11242d2cfbf4.00000000000000bc:head v 64'241, mlcod=64'239)", 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:54.557673+0000", 2024-09-11T05:51:55.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.84817794099999999, 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033303209999999998, 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557673+0000", 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557673+0000", 2024-09-11T05:51:55.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557675+0000", 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5099999999999999e-06 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557694+0000", 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9593999999999999e-05 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557715+0000", 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0814000000000001e-05 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557724+0000", 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5629999999999998e-06 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557797+0000", 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2470000000000002e-05 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.557815+0000", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7575999999999999e-05 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.560944+0000", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003129143 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.561003+0000", 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9651000000000002e-05 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:89803 3.1e e64/23 3:79bfa39c:::rbd_data.11242d2cfbf4.0000000000000090:head v 64'199, mlcod=62'197)", 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.71522637899999997, 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035203410000000002, 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690627+0000", 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690646+0000", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9173000000000001e-05 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690686+0000", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9541e-05 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690707+0000", 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0988999999999998e-05 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.691159+0000", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000452201 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.691210+0000", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1048000000000003e-05 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.694037+0000", 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002826402 2024-09-11T05:51:55.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.694145+0000", 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010836399999999999 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90436 3.1e e64/23 3:7990e080:::rbd_data.11242d2cfbf4.0000000000000086:head v 64'202, mlcod=64'200)", 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.40243853499999999, 2024-09-11T05:51:55.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035215329999999999, 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003414+0000", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6559999999999999e-06 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003507+0000", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3068999999999999e-05 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003525+0000", 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7363999999999998e-05 2024-09-11T05:51:55.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003534+0000", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0240000000000003e-06 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003607+0000", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3449999999999996e-05 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003628+0000", 2024-09-11T05:51:55.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509e-05 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.006878+0000", 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032491740000000001 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.006934+0000", 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6286999999999998e-05 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.497 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90501 3.0 e64/56 3:00050be8:::rbd_data.11242d2cfbf4.00000000000000ed:head v 64'244, mlcod=64'242)", 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.179448+0000", 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.22640344800000001, 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003279152, 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:55.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179448+0000", 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179448+0000", 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179450+0000", 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.114e-06 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179474+0000", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4564999999999999e-05 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179508+0000", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3325999999999999e-05 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179521+0000", 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3154e-05 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179597+0000", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5858000000000005e-05 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.179622+0000", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5287999999999999e-05 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.182663+0000", 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030409949999999999 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:55.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.182727+0000", 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3851999999999998e-05 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:55.502 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:3320 write 0x67b69cb thru 0x67c062b (0x9c61 bytes) 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:3321 read 0x821873a thru 0x8219c2d (0x14f4 bytes) 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:3322 read 0x7761f6e thru 0x776b377 (0x940a bytes) 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:3327 read 0x843ed52 thru 0x8443513 (0x47c2 bytes) 2024-09-11T05:51:55.504 INFO:teuthology.orchestra.run.smithi165.stdout:3328 trunc from 0xe23f33c to 0x2ead741 2024-09-11T05:51:55.505 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:55.504+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:55.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:55.523 INFO:teuthology.orchestra.run.smithi165.stdout:3329 trunc from 0x2ead741 to 0xe7dd50f 2024-09-11T05:51:55.533 INFO:teuthology.orchestra.run.smithi165.stdout:3330 write 0x7e7b7f thru 0x7ec94f (0x4dd1 bytes) 2024-09-11T05:51:55.585 INFO:teuthology.orchestra.run.smithi165.stdout:3331 write 0x2315620 thru 0x2322754 (0xd135 bytes) 2024-09-11T05:51:55.593 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.593 INFO:teuthology.orchestra.run.smithi165.stdout:3335 read 0x35534ad thru 0x355bead (0x8a01 bytes) 2024-09-11T05:51:55.593 INFO:teuthology.orchestra.run.smithi165.stdout:3336 write 0x5eac1ba thru 0x5eb3d2a (0x7b71 bytes) 2024-09-11T05:51:55.598 INFO:teuthology.orchestra.run.smithi165.stdout:3337 write 0xda529ca thru 0xda570cf (0x4706 bytes) 2024-09-11T05:51:55.604 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:55.600+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:55.606 INFO:teuthology.orchestra.run.smithi165.stdout:3338 writesame 0xdae74f8 thru 0xdaf4001 data_size 0x3(0xcb0a bytes) 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:3340 read 0x60513cc thru 0x605fec9 (0xeafe bytes) 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:3342 punch from 0x4c377d2 to 0x4c3bdb1, (0x45df bytes) 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:3343 read 0xde937fd thru 0xde9b269 (0x7a6d bytes) 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.627 INFO:teuthology.orchestra.run.smithi165.stdout:3345 write 0x3c0e0fc thru 0x3c16dce (0x8cd3 bytes) 2024-09-11T05:51:55.645 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:55.645 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:55.645 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:55.645 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:55.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:51:55.670 INFO:teuthology.orchestra.run.smithi165.stdout:3347 trunc from 0xe7dd50f to 0x1e57350 2024-09-11T05:51:55.696 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.696 INFO:teuthology.orchestra.run.smithi165.stdout:3350 read 0xb3782e thru 0xb433fa (0xbbcd bytes) 2024-09-11T05:51:55.705 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:55.704+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:55.727 INFO:teuthology.orchestra.run.smithi165.stdout:3351 writesame 0xa4ee452 thru 0xa4f3929 data_size 0x3c(0x54d8 bytes) 2024-09-11T05:51:55.727 INFO:teuthology.orchestra.run.smithi165.stdout:3352 writesame 0x94051a8 thru 0x940b72c data_size 0x1(0x6585 bytes) 2024-09-11T05:51:55.759 INFO:teuthology.orchestra.run.smithi165.stdout:3356 trunc from 0xa4f392a to 0xc9c3fb8 2024-09-11T05:51:55.768 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.771 INFO:teuthology.orchestra.run.smithi165.stdout:3358 trunc from 0xc9c3fb8 to 0xda03a65 2024-09-11T05:51:55.801 INFO:teuthology.orchestra.run.smithi165.stdout:3359 trunc from 0xda03a65 to 0x5411c12 2024-09-11T05:51:55.805 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:55.803+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:55.829 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:55.829 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:55.829 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:55.829 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:55.829 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:55.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:51:55.857 INFO:teuthology.orchestra.run.smithi165.stdout:3360 writesame 0xec6edf6 thru 0xec7745e data_size 0x1(0x8669 bytes) 2024-09-11T05:51:55.871 INFO:teuthology.orchestra.run.smithi165.stdout:3365 trunc from 0xec7745f to 0x9a7bb3e 2024-09-11T05:51:55.884 INFO:teuthology.orchestra.run.smithi165.stdout:3368 writesame 0x85a7f8a thru 0x85a9046 data_size 0x1(0x10bd bytes) 2024-09-11T05:51:55.897 INFO:teuthology.orchestra.run.smithi165.stdout:3369 write 0x65f0dce thru 0x65f86bd (0x78f0 bytes) 2024-09-11T05:51:55.909 INFO:teuthology.orchestra.run.smithi165.stdout:3370 write 0x8dc7b9f thru 0x8dd3a12 (0xbe74 bytes) 2024-09-11T05:51:55.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:51:55.933 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:55.931+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:55.936 INFO:teuthology.orchestra.run.smithi165.stdout:3371 writesame 0xbf3bec4 thru 0xbf4614e data_size 0x1(0xa28b bytes) 2024-09-11T05:51:55.948 INFO:teuthology.orchestra.run.smithi165.stdout:3372 writesame 0xd471c54 thru 0xd47fedf data_size 0x6c(0xe28c bytes) 2024-09-11T05:51:55.958 INFO:teuthology.orchestra.run.smithi165.stdout:3373 trunc from 0xd47fee0 to 0x7f7695c 2024-09-11T05:51:55.979 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:55.994 INFO:teuthology.orchestra.run.smithi165.stdout:3375 write 0xa490d13 thru 0xa49b0f0 (0xa3de bytes) 2024-09-11T05:51:56.006 INFO:teuthology.orchestra.run.smithi165.stdout:3376 punch from 0x4f04351 to 0x4f0e7aa, (0xa459 bytes) 2024-09-11T05:51:56.006 INFO:teuthology.orchestra.run.smithi165.stdout:3378 punch from 0x2dd6912 to 0x2de004d, (0x973b bytes) 2024-09-11T05:51:56.006 INFO:teuthology.orchestra.run.smithi165.stdout:3379 read 0x8c03171 thru 0x8c0e2d7 (0xb167 bytes) 2024-09-11T05:51:56.007 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:56.008+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.915762444999999, 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:51:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:56.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.674175062, 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.673564236000001, 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:51:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.673539828999999, 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:51:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:51:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.673529511, 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670925896, 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:51:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670915453999999, 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670903747000001, 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670892667, 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:56.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:51:56.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:51:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670857571999999, 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:51:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670817066, 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670783269999999, 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670744366999999, 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:51:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670657865999999, 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:51:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:51:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670640604999999, 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:51:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670334531, 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:51:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670274758, 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670253115, 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.670231912, 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:51:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.669187121, 2024-09-11T05:51:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:56.109 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:56.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:51:56.112 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:56.107+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3380 trunc from 0xa49b0f1 to 0x2f6391b 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3381 read 0x15c4d8a thru 0x15c667e (0x18f5 bytes) 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3382 punch from 0x112a7e2 to 0x112c9f8, (0x2216 bytes) 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3383 write 0x2a74ee3 thru 0x2a7a082 (0x51a0 bytes) 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3384 writesame 0x2f4dd74 thru 0x2f5133e data_size 0x1(0x35cb bytes) 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3385 punch from 0xffec1a to 0x1002f73, (0x4359 bytes) 2024-09-11T05:51:56.112 INFO:teuthology.orchestra.run.smithi165.stdout:3386 flatten 2024-09-11T05:51:56.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:51:56.187 INFO:teuthology.orchestra.run.smithi165.stdout:3388 write 0xa11efc0 thru 0xa1253bc (0x63fd bytes) 2024-09-11T05:51:56.203 INFO:teuthology.orchestra.run.smithi165.stdout:3390 trunc from 0xa1253bd to 0xbe8b5ec 2024-09-11T05:51:56.212 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:56.207+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:56.216 INFO:teuthology.orchestra.run.smithi165.stdout:3391 punch from 0x490bd5c to 0x49172f5, (0xb599 bytes) 2024-09-11T05:51:56.216 INFO:teuthology.orchestra.run.smithi165.stdout:3392 read 0x145d98b thru 0x145f2aa (0x1920 bytes) 2024-09-11T05:51:56.216 INFO:teuthology.orchestra.run.smithi165.stdout:3394 writesame 0xbe18a7 thru 0xbeca79 data_size 0x1(0xb1d3 bytes) 2024-09-11T05:51:56.242 INFO:teuthology.orchestra.run.smithi165.stdout:3396 writesame 0x8e52e2d thru 0x8e61a6d data_size 0x1(0xec41 bytes) 2024-09-11T05:51:56.247 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:56.247 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:51:56.247 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:51:56.247 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:56.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:51:56.277 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:56.277 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:56.294 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:56.294 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:56.312 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:56.312 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:56.312 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:56.307+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:56.327 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:51:56.332 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:56.332 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:56.336 INFO:teuthology.orchestra.run.smithi165.stdout:3397 trunc from 0xbe8b5ec to 0x5fb3e62 2024-09-11T05:51:56.355 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:56.356 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:56.371 INFO:teuthology.orchestra.run.smithi165.stdout:3398 write 0x2166d2c thru 0x216e7f0 (0x7ac5 bytes) 2024-09-11T05:51:56.379 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:56.379 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:56.384 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.401 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:56.401 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:56.402 INFO:teuthology.orchestra.run.smithi165.stdout:3400 writesame 0xa2fa79f thru 0xa305d2f data_size 0x1(0xb591 bytes) 2024-09-11T05:51:56.410 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:56.411 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:56.413 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:56.412+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:56.423 INFO:teuthology.orchestra.run.smithi165.stdout:3401 write 0xe2f151f thru 0xe2fbc8c (0xa76e bytes) 2024-09-11T05:51:56.423 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:56.424 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:56.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:51:56.436 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.436 INFO:teuthology.orchestra.run.smithi165.stdout:3403 read 0xd28bc48 thru 0xd292120 (0x64d9 bytes) 2024-09-11T05:51:56.436 INFO:teuthology.orchestra.run.smithi165.stdout:3404 read 0x6d1db76 thru 0x6d22761 (0x4bec bytes) 2024-09-11T05:51:56.436 INFO:teuthology.orchestra.run.smithi165.stdout:3406 write 0xbefa912 thru 0xbefde44 (0x3533 bytes) 2024-09-11T05:51:56.444 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.444 INFO:teuthology.orchestra.run.smithi165.stdout:3409 write 0xab765c6 thru 0xab7b606 (0x5041 bytes) 2024-09-11T05:51:56.466 INFO:teuthology.orchestra.run.smithi165.stdout:3410 trunc from 0xe2fbc8d to 0x5533983 2024-09-11T05:51:56.500 INFO:teuthology.orchestra.run.smithi165.stdout:3411 trunc from 0x5533983 to 0xbf14cd4 2024-09-11T05:51:56.514 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:56.512+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:56.528 INFO:teuthology.orchestra.run.smithi165.stdout:3412 writesame 0xe592d47 thru 0xe59fe61 data_size 0x1(0xd11b bytes) 2024-09-11T05:51:56.544 INFO:teuthology.orchestra.run.smithi165.stdout:3413 trunc from 0xe59fe62 to 0x3a4282f 2024-09-11T05:51:56.559 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:56.560 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:56.571 INFO:teuthology.orchestra.run.smithi165.stdout:3414 write 0x43b2a1b thru 0x43c0c7c (0xe262 bytes) 2024-09-11T05:51:56.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:51:56.588 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.590 INFO:teuthology.orchestra.run.smithi165.stdout:3416 trunc from 0x43c0c7d to 0x5037907 2024-09-11T05:51:56.620 INFO:teuthology.orchestra.run.smithi165.stdout:3417 writesame 0x754d82e thru 0x755beb6 data_size 0x1(0xe689 bytes) 2024-09-11T05:51:56.621 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:56.620+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:56.629 INFO:teuthology.orchestra.run.smithi165.stdout:3418 trunc from 0x755beb7 to 0x26baa9b 2024-09-11T05:51:56.682 INFO:teuthology.orchestra.run.smithi165.stdout:3419 writesame 0xb14c421 thru 0xb151574 data_size 0x1e(0x5154 bytes) 2024-09-11T05:51:56.682 INFO:teuthology.orchestra.run.smithi165.stdout:3421 read 0x331404c thru 0x331526e (0x1223 bytes) 2024-09-11T05:51:56.682 INFO:teuthology.orchestra.run.smithi165.stdout:3422 punch from 0x3f21d1c to 0x3f30d76, (0xf05a bytes) 2024-09-11T05:51:56.682 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.682 INFO:teuthology.orchestra.run.smithi165.stdout:3425 write 0x2f61953 thru 0x2f629db (0x1089 bytes) 2024-09-11T05:51:56.689 INFO:teuthology.orchestra.run.smithi165.stdout:3426 read 0x2a47e84 thru 0x2a542c0 (0xc43d bytes) 2024-09-11T05:51:56.689 INFO:teuthology.orchestra.run.smithi165.stdout:3427 writesame 0x1d08415 thru 0x1d18190 data_size 0x2(0xfd7c bytes) 2024-09-11T05:51:56.715 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:51:56.711+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:51:56.719 INFO:teuthology.orchestra.run.smithi165.stdout:3429 write 0x58a333f thru 0x58a7bcf (0x4891 bytes) 2024-09-11T05:51:56.721 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:56.721 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:56.730 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:51:56.741 INFO:teuthology.orchestra.run.smithi165.stdout:3433 trunc from 0xb151575 to 0x7f3d23e 2024-09-11T05:51:56.764 INFO:teuthology.orchestra.run.smithi165.stdout:3434 trunc from 0x7f3d23e to 0x31bf0cb 2024-09-11T05:51:56.816 INFO:teuthology.orchestra.run.smithi165.stdout:3436 write 0x961530c thru 0x961c8d3 (0x75c8 bytes) 2024-09-11T05:51:56.816 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:56.811+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:56.829 INFO:teuthology.orchestra.run.smithi165.stdout:3437 trunc from 0x961c8d4 to 0x80378e8 2024-09-11T05:51:56.877 INFO:teuthology.orchestra.run.smithi165.stdout:3440 writesame 0xc42c0ef thru 0xc437a89 data_size 0x55(0xb99b bytes) 2024-09-11T05:51:56.877 INFO:teuthology.orchestra.run.smithi165.stdout:3442 write 0x4fa601d thru 0x4fab109 (0x50ed bytes) 2024-09-11T05:51:56.887 INFO:teuthology.orchestra.run.smithi165.stdout:3443 punch from 0xb19c3f9 to 0xb1a7187, (0xad8e bytes) 2024-09-11T05:51:56.887 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.894 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:56.894 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:56.905 INFO:teuthology.orchestra.run.smithi165.stdout:3447 write 0xd901f2d thru 0xd910706 (0xe7da bytes) 2024-09-11T05:51:56.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:51:56.917 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:56.915+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:56.926 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.926 INFO:teuthology.orchestra.run.smithi165.stdout:3449 write 0x31952ba thru 0x31a060e (0xb355 bytes) 2024-09-11T05:51:56.939 INFO:teuthology.orchestra.run.smithi165.stdout:3450 read 0xa585524 thru 0xa593a70 (0xe54d bytes) 2024-09-11T05:51:56.942 INFO:teuthology.orchestra.run.smithi165.stdout:3452 trunc from 0xd910707 to 0xbda674f 2024-09-11T05:51:56.971 INFO:teuthology.orchestra.run.smithi165.stdout:3453 trunc from 0xbda674f to 0x8f42a9c 2024-09-11T05:51:56.986 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:56.993 INFO:teuthology.orchestra.run.smithi165.stdout:3455 trunc from 0x8f42a9c to 0xcec1ac5 2024-09-11T05:51:57.007 INFO:teuthology.orchestra.run.smithi165.stdout:3456 punch from 0x4d9d94a to 0x4da0c30, (0x32e6 bytes) 2024-09-11T05:51:57.007 INFO:teuthology.orchestra.run.smithi165.stdout:3459 read 0x25ba2d0 thru 0x25c6e04 (0xcb35 bytes) 2024-09-11T05:51:57.007 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.007 INFO:teuthology.orchestra.run.smithi165.stdout:3461 writesame 0x5fa41a5 thru 0x5fa88df data_size 0x7(0x473b bytes) 2024-09-11T05:51:57.016 INFO:teuthology.orchestra.run.smithi165.stdout:3464 write 0x9b35ce5 thru 0x9b3c5ef (0x690b bytes) 2024-09-11T05:51:57.017 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:57.016+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.029 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.030 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.032 INFO:teuthology.orchestra.run.smithi165.stdout:3466 trunc from 0xcec1ac5 to 0xaa53a8f 2024-09-11T05:51:57.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:51:57.046 INFO:teuthology.orchestra.run.smithi165.stdout:3467 read 0x665e354 thru 0x665f143 (0xdf0 bytes) 2024-09-11T05:51:57.046 INFO:teuthology.orchestra.run.smithi165.stdout:3468 punch from 0x92630a9 to 0x9270911, (0xd868 bytes) 2024-09-11T05:51:57.046 INFO:teuthology.orchestra.run.smithi165.stdout:3470 read 0x3868824 thru 0x3872dde (0xa5bb bytes) 2024-09-11T05:51:57.047 INFO:teuthology.orchestra.run.smithi165.stdout:3471 read 0x3034dd9 thru 0x303c557 (0x777f bytes) 2024-09-11T05:51:57.047 INFO:teuthology.orchestra.run.smithi165.stdout:3473 write 0x2ff64a4 thru 0x2ff7fac (0x1b09 bytes) 2024-09-11T05:51:57.053 INFO:teuthology.orchestra.run.smithi165.stdout:3474 read 0x5639770 thru 0x563b544 (0x1dd5 bytes) 2024-09-11T05:51:57.059 INFO:teuthology.orchestra.run.smithi165.stdout:3475 trunc from 0xaa53a8f to 0xd6e580f 2024-09-11T05:51:57.075 INFO:teuthology.orchestra.run.smithi165.stdout:3476 writesame 0x24111b6 thru 0x2414054 data_size 0x23(0x2e9f bytes) 2024-09-11T05:51:57.085 INFO:teuthology.orchestra.run.smithi165.stdout:3479 read 0x6270888 thru 0x627f66b (0xede4 bytes) 2024-09-11T05:51:57.085 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.088 INFO:teuthology.orchestra.run.smithi165.stdout:3481 trunc from 0xd6e580f to 0xbc985dc 2024-09-11T05:51:57.103 INFO:teuthology.orchestra.run.smithi165.stdout:3482 read 0x2fa8b77 thru 0x2fab4c1 (0x294b bytes) 2024-09-11T05:51:57.103 INFO:teuthology.orchestra.run.smithi165.stdout:3483 punch from 0x1471f2b to 0x1479390, (0x7465 bytes) 2024-09-11T05:51:57.103 INFO:teuthology.orchestra.run.smithi165.stdout:3487 punch from 0x40d4f95 to 0x40d8f63, (0x3fce bytes) 2024-09-11T05:51:57.103 INFO:teuthology.orchestra.run.smithi165.stdout:3488 write 0x1a1712a thru 0x1a1949f (0x2376 bytes) 2024-09-11T05:51:57.113 INFO:teuthology.orchestra.run.smithi165.stdout:3490 write 0x76099d7 thru 0x760afd3 (0x15fd bytes) 2024-09-11T05:51:57.118 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:57.116+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:57.120 INFO:teuthology.orchestra.run.smithi165.stdout:3491 read 0xb7b83b thru 0xb8862a (0xcdf0 bytes) 2024-09-11T05:51:57.122 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.122 INFO:teuthology.orchestra.run.smithi165.stdout:3496 write 0x636267d thru 0x636b321 (0x8ca5 bytes) 2024-09-11T05:51:57.134 INFO:teuthology.orchestra.run.smithi165.stdout:3498 write 0x619141a thru 0x6199bf8 (0x87df bytes) 2024-09-11T05:51:57.168 INFO:teuthology.orchestra.run.smithi165.stdout:3499 writesame 0xdd426ff thru 0xdd430c7 data_size 0x1(0x9c9 bytes) 2024-09-11T05:51:57.168 INFO:teuthology.orchestra.run.smithi165.stdout:3500 read 0x1308aec thru 0x130c6bd (0x3bd2 bytes) 2024-09-11T05:51:57.168 INFO:teuthology.orchestra.run.smithi165.stdout:3504 read 0xcc204e6 thru 0xcc2776e (0x7289 bytes) 2024-09-11T05:51:57.172 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.172 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:51:57.199 INFO:teuthology.orchestra.run.smithi165.stdout:3505 trunc from 0xdd430c8 to 0x7316e9 2024-09-11T05:51:57.244 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:57.244+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:57.251 INFO:teuthology.orchestra.run.smithi165.stdout:3507 punch from 0x1d5760 to 0x1dabe4, (0x5484 bytes) 2024-09-11T05:51:57.266 INFO:teuthology.orchestra.run.smithi165.stdout:3508 write 0x1970e38 thru 0x197a77e (0x9947 bytes) 2024-09-11T05:51:57.278 INFO:teuthology.orchestra.run.smithi165.stdout:3509 clone 22 order 24 su 131072 sc 3 2024-09-11T05:51:57.320 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:57.315+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:57.330 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.330 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:51:57.420 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:57.415+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:57.521 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:57.520+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:57.528 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.528 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:51:57.562 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-09-11T05:51:57.598 INFO:teuthology.orchestra.run.smithi165.stdout:3510 write 0xd282c18 thru 0xd28eebe (0xc2a7 bytes) 2024-09-11T05:51:57.606 INFO:teuthology.orchestra.run.smithi165.stdout:3512 punch from 0x6476f17 to 0x648467c, (0xd765 bytes) 2024-09-11T05:51:57.606 INFO:teuthology.orchestra.run.smithi165.stdout:3513 punch from 0x19449ee to 0x1952d4f, (0xe361 bytes) 2024-09-11T05:51:57.606 INFO:teuthology.orchestra.run.smithi165.stdout:3514 write 0x211e8d8 thru 0x2128e1f (0xa548 bytes) 2024-09-11T05:51:57.622 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:57.619+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:57.670 INFO:teuthology.orchestra.run.smithi165.stdout:3515 read 0x1f84a5b thru 0x1f89752 (0x4cf8 bytes) 2024-09-11T05:51:57.672 INFO:teuthology.orchestra.run.smithi165.stdout:3516 punch from 0x5b235c4 to 0x5b31f75, (0xe9b1 bytes) 2024-09-11T05:51:57.672 INFO:teuthology.orchestra.run.smithi165.stdout:3517 writesame 0x6f484c7 thru 0x6f5641b data_size 0x1(0xdf55 bytes) 2024-09-11T05:51:57.700 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.701 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:51:57.745 INFO:teuthology.orchestra.run.smithi165.stdout:3518 write 0x5d58be8 thru 0x5d62590 (0x99a9 bytes) 2024-09-11T05:51:57.746 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:57.744+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:57.761 INFO:teuthology.orchestra.run.smithi165.stdout:3519 writesame 0xd467b70 thru 0xd471f83 data_size 0x2(0xa414 bytes) 2024-09-11T05:51:57.763 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.763 INFO:teuthology.orchestra.run.smithi165.stdout:3523 writesame 0x1630be2 thru 0x1635bed data_size 0x2f(0x500c bytes) 2024-09-11T05:51:57.768 INFO:teuthology.orchestra.run.smithi165.stdout:3524 trunc from 0xd471f84 to 0xe390a36 2024-09-11T05:51:57.777 INFO:teuthology.orchestra.run.smithi165.stdout:3525 write 0xc87dcf5 thru 0xc8872f3 (0x95ff bytes) 2024-09-11T05:51:57.783 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.783 INFO:teuthology.orchestra.run.smithi165.stdout:3527 write 0x9f14ae2 thru 0x9f1d122 (0x8641 bytes) 2024-09-11T05:51:57.789 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:57.789 INFO:teuthology.orchestra.run.smithi165.stdout:3530 punch from 0x6cfe4dc to 0x6d00b7f, (0x26a3 bytes) 2024-09-11T05:51:57.813 INFO:teuthology.orchestra.run.smithi165.stdout:3531 trunc from 0xe390a36 to 0xaeb537 2024-09-11T05:51:57.823 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:57.824+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:57.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:51:57.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:51:57.893 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-11T05:51:57.893 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T05:51:57.893 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T05:51:57.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-11T05:51:57.907 INFO:teuthology.orchestra.run.smithi165.stdout:3532 write 0xe5ed1c4 thru 0xe5f87c5 (0xb602 bytes) 2024-09-11T05:51:57.911 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:57.911 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:57.914 INFO:teuthology.orchestra.run.smithi165.stdout:3533 punch from 0x59a56b4 to 0x59abae8, (0x6434 bytes) 2024-09-11T05:51:57.914 INFO:teuthology.orchestra.run.smithi165.stdout:3534 write 0xa180d6c thru 0xa18ff2b (0xf1c0 bytes) 2024-09-11T05:51:57.923 INFO:teuthology.orchestra.run.smithi165.stdout:3535 writesame 0x3f42679 thru 0x3f4e5d7 data_size 0x1(0xbf5f bytes) 2024-09-11T05:51:57.924 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:57.924+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:57.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:51:57.958 INFO:teuthology.orchestra.run.smithi165.stdout:3536 write 0x5b1af62 thru 0x5b249ac (0x9a4b bytes) 2024-09-11T05:51:57.965 INFO:teuthology.orchestra.run.smithi165.stdout:3538 writesame 0x289688b thru 0x28a1f77 data_size 0x1(0xb6ed bytes) 2024-09-11T05:51:58.016 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.016 INFO:teuthology.orchestra.run.smithi165.stdout:3540 read 0x473a546 thru 0x4743ffd (0x9ab8 bytes) 2024-09-11T05:51:58.021 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:51:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T05:51:58.021 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:51:58.025 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:58.019+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:58.031 INFO:teuthology.orchestra.run.smithi165.stdout:3541 trunc from 0xe5f87c6 to 0x47786da 2024-09-11T05:51:58.060 INFO:teuthology.orchestra.run.smithi165.stdout:3542 writesame 0x4b9d285 thru 0x4ba772e data_size 0x7(0xa4aa bytes) 2024-09-11T05:51:58.063 INFO:teuthology.orchestra.run.smithi165.stdout:3544 punch from 0x28006f6 to 0x2805e56, (0x5760 bytes) 2024-09-11T05:51:58.063 INFO:teuthology.orchestra.run.smithi165.stdout:3545 read 0x29f3e1b thru 0x29f824f (0x4435 bytes) 2024-09-11T05:51:58.074 INFO:teuthology.orchestra.run.smithi165.stdout:3546 trunc from 0x4ba772f to 0xa00dcae 2024-09-11T05:51:58.083 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.085 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:58.085 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:58.097 INFO:teuthology.orchestra.run.smithi165.stdout:3548 writesame 0xc0ae964 thru 0xc0b0110 data_size 0x13(0x17ad bytes) 2024-09-11T05:51:58.097 INFO:teuthology.orchestra.run.smithi165.stdout:3549 writesame 0x50c347e thru 0x50cb7ac data_size 0x1(0x832f bytes) 2024-09-11T05:51:58.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:51:58.117 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.117 INFO:teuthology.orchestra.run.smithi165.stdout:3553 write 0x1ccdade thru 0x1cdab59 (0xd07c bytes) 2024-09-11T05:51:58.119 INFO:teuthology.orchestra.run.smithi165.stdout:3555 punch from 0x1a548cb to 0x1a59ae8, (0x521d bytes) 2024-09-11T05:51:58.119 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.125 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:58.124+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:58.132 INFO:teuthology.orchestra.run.smithi165.stdout:3559 writesame 0xdee397d thru 0xdef20d5 data_size 0x67(0xe759 bytes) 2024-09-11T05:51:58.132 INFO:teuthology.orchestra.run.smithi165.stdout:3562 punch from 0xa32fa19 to 0xa33705e, (0x7645 bytes) 2024-09-11T05:51:58.147 INFO:teuthology.orchestra.run.smithi165.stdout:3563 trunc from 0xdef20d6 to 0x794366f 2024-09-11T05:51:58.158 INFO:teuthology.orchestra.run.smithi165.stdout:3566 read 0x3bfe40e thru 0x3c04cfe (0x68f1 bytes) 2024-09-11T05:51:58.176 INFO:teuthology.orchestra.run.smithi165.stdout:3567 write 0x8c63149 thru 0x8c655ec (0x24a4 bytes) 2024-09-11T05:51:58.183 INFO:teuthology.orchestra.run.smithi165.stdout:3570 writesame 0x174266e thru 0x1751af3 data_size 0x3(0xf486 bytes) 2024-09-11T05:51:58.202 INFO:teuthology.orchestra.run.smithi165.stdout:3571 punch from 0x7e62412 to 0x7e63590, (0x117e bytes) 2024-09-11T05:51:58.202 INFO:teuthology.orchestra.run.smithi165.stdout:3572 punch from 0x377b7f7 to 0x37840c4, (0x88cd bytes) 2024-09-11T05:51:58.224 INFO:teuthology.orchestra.run.smithi165.stdout:3574 write 0xb976fa9 thru 0xb97bafb (0x4b53 bytes) 2024-09-11T05:51:58.227 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:58.223+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:58.231 INFO:teuthology.orchestra.run.smithi165.stdout:3575 read 0x3859ac thru 0x388ebe (0x3513 bytes) 2024-09-11T05:51:58.232 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.232 INFO:teuthology.orchestra.run.smithi165.stdout:3577 write 0x196f4a5 thru 0x1974ed4 (0x5a30 bytes) 2024-09-11T05:51:58.236 INFO:teuthology.orchestra.run.smithi165.stdout:3578 write 0x964bb47 thru 0x965894e (0xce08 bytes) 2024-09-11T05:51:58.239 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:58.239 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:58.243 INFO:teuthology.orchestra.run.smithi165.stdout:3579 write 0xb7283c1 thru 0xb72d29f (0x4edf bytes) 2024-09-11T05:51:58.246 INFO:teuthology.orchestra.run.smithi165.stdout:3580 writesame 0x6b64d5c thru 0x6b65ffd data_size 0xa(0x12a2 bytes) 2024-09-11T05:51:58.249 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.249 INFO:teuthology.orchestra.run.smithi165.stdout:3582 write 0x218d750 thru 0x218f0cc (0x197d bytes) 2024-09-11T05:51:58.252 INFO:teuthology.orchestra.run.smithi165.stdout:3583 writesame 0x9d4bd07 thru 0x9d4d7ef data_size 0x1(0x1ae9 bytes) 2024-09-11T05:51:58.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:51:58.275 INFO:teuthology.orchestra.run.smithi165.stdout:3584 trunc from 0xb97bafc to 0x51aa8fa 2024-09-11T05:51:58.337 INFO:teuthology.orchestra.run.smithi165.stdout:3585 write 0xeabee0f thru 0xeabf287 (0x479 bytes) 2024-09-11T05:51:58.344 INFO:teuthology.orchestra.run.smithi165.stdout:3590 writesame 0x92969f thru 0x93585a data_size 0x6(0xc1bc bytes) 2024-09-11T05:51:58.349 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:58.347+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:58.352 INFO:teuthology.orchestra.run.smithi165.stdout:3591 write 0x469fb29 thru 0x46a858f (0x8a67 bytes) 2024-09-11T05:51:58.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:51:58.381 INFO:teuthology.orchestra.run.smithi165.stdout:3593 trunc from 0xeabf288 to 0x4e31891 2024-09-11T05:51:58.418 INFO:teuthology.orchestra.run.smithi165.stdout:3594 writesame 0x8b79fbe thru 0x8b81af9 data_size 0x42(0x7b3c bytes) 2024-09-11T05:51:58.418 INFO:teuthology.orchestra.run.smithi165.stdout:3595 write 0x14e19de thru 0x14eb571 (0x9b94 bytes) 2024-09-11T05:51:58.420 INFO:teuthology.orchestra.run.smithi165.stdout:3596 write 0x7deea8a thru 0x7df121f (0x2796 bytes) 2024-09-11T05:51:58.426 INFO:teuthology.orchestra.run.smithi165.stdout:3597 writesame 0x3664ea9 thru 0x3673e50 data_size 0x4(0xefa8 bytes) 2024-09-11T05:51:58.427 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:51:58.423+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:51:58.430 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:58.431 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:58.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:51:58.445 INFO:teuthology.orchestra.run.smithi165.stdout:3598 read 0x4ed31a9 thru 0x4ed83be (0x5216 bytes) 2024-09-11T05:51:58.445 INFO:teuthology.orchestra.run.smithi165.stdout:3599 read 0x14033a6 thru 0x140553a (0x2195 bytes) 2024-09-11T05:51:58.445 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.445 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.445 INFO:teuthology.orchestra.run.smithi165.stdout:3603 read 0x2329c85 thru 0x2330de8 (0x7164 bytes) 2024-09-11T05:51:58.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:51:58.464 INFO:teuthology.orchestra.run.smithi165.stdout:3604 writesame 0xdcb0b5a thru 0xdcb32f4 data_size 0x1(0x279b bytes) 2024-09-11T05:51:58.464 INFO:teuthology.orchestra.run.smithi165.stdout:3605 writesame 0xce2755f thru 0xce31d87 data_size 0x1(0xa829 bytes) 2024-09-11T05:51:58.500 INFO:teuthology.orchestra.run.smithi165.stdout:3606 writesame 0xe458263 thru 0xe464a90 data_size 0x49(0xc82e bytes) 2024-09-11T05:51:58.502 INFO:teuthology.orchestra.run.smithi165.stdout:3607 write 0xd54a72b thru 0xd55908a (0xe960 bytes) 2024-09-11T05:51:58.505 INFO:teuthology.orchestra.run.smithi165.stdout:3608 trunc from 0xe464a91 to 0xe852659 2024-09-11T05:51:58.511 INFO:teuthology.orchestra.run.smithi165.stdout:3609 write 0x121321b thru 0x121fe4b (0xcc31 bytes) 2024-09-11T05:51:58.514 INFO:teuthology.orchestra.run.smithi165.stdout:3611 punch from 0x3fe429e to 0x3fe99a5, (0x5707 bytes) 2024-09-11T05:51:58.514 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.514 INFO:teuthology.orchestra.run.smithi165.stdout:3614 punch from 0xe7f7414 to 0xe7f7e5a, (0xa46 bytes) 2024-09-11T05:51:58.514 INFO:teuthology.orchestra.run.smithi165.stdout:3616 write 0x12e168d thru 0x12e7b1c (0x6490 bytes) 2024-09-11T05:51:58.515 INFO:teuthology.orchestra.run.smithi165.stdout:3617 write 0x2177a06 thru 0x2179fcf (0x25ca bytes) 2024-09-11T05:51:58.518 INFO:teuthology.orchestra.run.smithi165.stdout:3619 writesame 0x8ec017d thru 0x8ecf09e data_size 0x39(0xef22 bytes) 2024-09-11T05:51:58.521 INFO:teuthology.orchestra.run.smithi165.stdout:3620 read 0x8a69f67 thru 0x8a6ce8a (0x2f24 bytes) 2024-09-11T05:51:58.521 INFO:teuthology.orchestra.run.smithi165.stdout:3621 write 0xe0db823 thru 0xe0e434a (0x8b28 bytes) 2024-09-11T05:51:58.525 INFO:teuthology.orchestra.run.smithi165.stdout:3622 write 0x392820b thru 0x3935720 (0xd516 bytes) 2024-09-11T05:51:58.527 INFO:teuthology.orchestra.run.smithi165.stdout:3623 write 0xdcea387 thru 0xdcf2fb4 (0x8c2e bytes) 2024-09-11T05:51:58.530 INFO:teuthology.orchestra.run.smithi165.stdout:3625 punch from 0xadc4aad to 0xadcbb0e, (0x7061 bytes) 2024-09-11T05:51:58.530 INFO:teuthology.orchestra.run.smithi165.stdout:3627 writesame 0x133fb95 thru 0x134bb47 data_size 0x41(0xbfb3 bytes) 2024-09-11T05:51:58.533 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.533 INFO:teuthology.orchestra.run.smithi165.stdout:3630 writesame 0xd517217 thru 0xd520097 data_size 0x1(0x8e81 bytes) 2024-09-11T05:51:58.537 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:58.535+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:58.549 INFO:teuthology.orchestra.run.smithi165.stdout:3631 write 0x6055c6b thru 0x605ec05 (0x8f9b bytes) 2024-09-11T05:51:58.552 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.552 INFO:teuthology.orchestra.run.smithi165.stdout:3634 write 0x5e7173f thru 0x5e8031c (0xebde bytes) 2024-09-11T05:51:58.554 INFO:teuthology.orchestra.run.smithi165.stdout:3635 read 0xc96f665 thru 0xc97acf2 (0xb68e bytes) 2024-09-11T05:51:58.554 INFO:teuthology.orchestra.run.smithi165.stdout:3637 write 0xc8e1894 thru 0xc8e5c40 (0x43ad bytes) 2024-09-11T05:51:58.556 INFO:teuthology.orchestra.run.smithi165.stdout:3640 read 0x858bdda thru 0x8592ab8 (0x6cdf bytes) 2024-09-11T05:51:58.556 INFO:teuthology.orchestra.run.smithi165.stdout:3641 read 0x173984b thru 0x17450a7 (0xb85d bytes) 2024-09-11T05:51:58.557 INFO:teuthology.orchestra.run.smithi165.stdout:3642 read 0x8f50517 thru 0x8f53382 (0x2e6c bytes) 2024-09-11T05:51:58.557 INFO:teuthology.orchestra.run.smithi165.stdout:3643 read 0x135663c thru 0x1362775 (0xc13a bytes) 2024-09-11T05:51:58.558 INFO:teuthology.orchestra.run.smithi165.stdout:3645 writesame 0x3516359 thru 0x3521d1c data_size 0x12(0xb9c4 bytes) 2024-09-11T05:51:58.561 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.561 INFO:teuthology.orchestra.run.smithi165.stdout:3647 writesame 0xcedaaa4 thru 0xcedb597 data_size 0x2(0xaf4 bytes) 2024-09-11T05:51:58.563 INFO:teuthology.orchestra.run.smithi165.stdout:3648 punch from 0xbdbe00f to 0xbdcae52, (0xce43 bytes) 2024-09-11T05:51:58.563 INFO:teuthology.orchestra.run.smithi165.stdout:3650 punch from 0xa138e0 to 0xa1edc1, (0xb4e1 bytes) 2024-09-11T05:51:58.581 INFO:teuthology.orchestra.run.smithi165.stdout:3651 trunc from 0xe852659 to 0x2eef859 2024-09-11T05:51:58.597 INFO:teuthology.orchestra.run.smithi165.stdout:3652 read 0x186de87 thru 0x1879e6c (0xbfe6 bytes) 2024-09-11T05:51:58.598 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.598 INFO:teuthology.orchestra.run.smithi165.stdout:3655 punch from 0x1b62516 to 0x1b62729, (0x213 bytes) 2024-09-11T05:51:58.598 INFO:teuthology.orchestra.run.smithi165.stdout:3657 punch from 0x1dc2253 to 0x1dcd172, (0xaf1f bytes) 2024-09-11T05:51:58.598 INFO:teuthology.orchestra.run.smithi165.stdout:3658 punch from 0x103cff9 to 0x10425f8, (0x55ff bytes) 2024-09-11T05:51:58.598 INFO:teuthology.orchestra.run.smithi165.stdout:3660 read 0x23f694 thru 0x248f24 (0x9891 bytes) 2024-09-11T05:51:58.599 INFO:teuthology.orchestra.run.smithi165.stdout:3661 read 0x29a0bf1 thru 0x29afb0b (0xef1b bytes) 2024-09-11T05:51:58.608 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:51:58.608 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:51:58.616 INFO:teuthology.orchestra.run.smithi165.stdout:3663 write 0x80b9bf0 thru 0x80bfd00 (0x6111 bytes) 2024-09-11T05:51:58.620 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:51:58.620 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:51:58.628 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:51:58.628 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:51:58.628 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:58.628+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:58.634 INFO:teuthology.orchestra.run.smithi165.stdout:3664 write 0xbef2d69 thru 0xbef9653 (0x68eb bytes) 2024-09-11T05:51:58.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:51:58.638 INFO:teuthology.orchestra.run.smithi165.stdout:3665 write 0x8ac5036 thru 0x8ad4598 (0xf563 bytes) 2024-09-11T05:51:58.640 INFO:teuthology.orchestra.run.smithi165.stdout:3666 punch from 0xab11154 to 0xab1d912, (0xc7be bytes) 2024-09-11T05:51:58.648 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:51:58.649 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:51:58.655 INFO:teuthology.orchestra.run.smithi165.stdout:3667 trunc from 0xbef9654 to 0x18dffb5 2024-09-11T05:51:58.667 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:51:58.667 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:51:58.672 INFO:teuthology.orchestra.run.smithi165.stdout:3669 read 0x1617aba thru 0x1624dfe (0xd345 bytes) 2024-09-11T05:51:58.673 INFO:teuthology.orchestra.run.smithi165.stdout:3670 punch from 0x73d937 to 0x73e264, (0x92d bytes) 2024-09-11T05:51:58.673 INFO:teuthology.orchestra.run.smithi165.stdout:3671 read 0x308663 thru 0x314f89 (0xc927 bytes) 2024-09-11T05:51:58.673 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.673 INFO:teuthology.orchestra.run.smithi165.stdout:3673 punch from 0x144f248 to 0x145b2ed, (0xc0a5 bytes) 2024-09-11T05:51:58.686 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:51:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:51:58.689 INFO:teuthology.orchestra.run.smithi165.stdout:3674 write 0x7b18e72 thru 0x7b1cfa5 (0x4134 bytes) 2024-09-11T05:51:58.708 INFO:teuthology.orchestra.run.smithi165.stdout:3675 write 0xd123a2f thru 0xd12490c (0xede bytes) 2024-09-11T05:51:58.712 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.712 INFO:teuthology.orchestra.run.smithi165.stdout:3679 write 0xb6fefb5 thru 0xb707f9a (0x8fe6 bytes) 2024-09-11T05:51:58.719 INFO:teuthology.orchestra.run.smithi165.stdout:3680 writesame 0x5e0d296 thru 0x5e171b2 data_size 0x4d(0x9f1d bytes) 2024-09-11T05:51:58.720 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:51:58.721 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:51:58.723 INFO:teuthology.orchestra.run.smithi165.stdout:3681 punch from 0x36d7f3f to 0x36d9f78, (0x2039 bytes) 2024-09-11T05:51:58.723 INFO:teuthology.orchestra.run.smithi165.stdout:3682 punch from 0x39b7e1f to 0x39c2332, (0xa513 bytes) 2024-09-11T05:51:58.723 INFO:teuthology.orchestra.run.smithi165.stdout:3683 write 0x9be1fd thru 0x9cc36c (0xe170 bytes) 2024-09-11T05:51:58.726 INFO:teuthology.orchestra.run.smithi165.stdout:3684 writesame 0xc17c13e thru 0xc183231 data_size 0x2(0x70f4 bytes) 2024-09-11T05:51:58.729 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:58.727+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:58.741 INFO:teuthology.orchestra.run.smithi165.stdout:3685 trunc from 0xd12490d to 0x8e5d126 2024-09-11T05:51:58.743 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:51:58.744 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:51:58.754 INFO:teuthology.orchestra.run.smithi165.stdout:3686 punch from 0x55ef95a to 0x55f33b9, (0x3a5f bytes) 2024-09-11T05:51:58.762 INFO:teuthology.orchestra.run.smithi165.stdout:3689 write 0x9e29640 thru 0x9e32643 (0x9004 bytes) 2024-09-11T05:51:58.763 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:51:58.763 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:51:58.769 INFO:teuthology.orchestra.run.smithi165.stdout:3691 read 0x8205151 thru 0x820a545 (0x53f5 bytes) 2024-09-11T05:51:58.775 INFO:teuthology.orchestra.run.smithi165.stdout:3693 writesame 0xa16bd4b thru 0xa1745b3 data_size 0x1(0x8869 bytes) 2024-09-11T05:51:58.775 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.775 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.779 INFO:teuthology.orchestra.run.smithi165.stdout:3699 trunc from 0xa1745b4 to 0x766125c 2024-09-11T05:51:58.791 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:58.792 INFO:teuthology.orchestra.run.smithi165.stdout:3702 read 0x5d22e0a thru 0x5d23db0 (0xfa7 bytes) 2024-09-11T05:51:58.792 INFO:teuthology.orchestra.run.smithi165.stdout:3703 read 0x297e66d thru 0x2980a90 (0x2424 bytes) 2024-09-11T05:51:58.792 INFO:teuthology.orchestra.run.smithi165.stdout:3705 read 0x2916140 thru 0x29207e9 (0xa6aa bytes) 2024-09-11T05:51:58.792 INFO:teuthology.orchestra.run.smithi165.stdout:3706 read 0x57522f2 thru 0x5760483 (0xe192 bytes) 2024-09-11T05:51:58.805 INFO:teuthology.orchestra.run.smithi165.stdout:3707 writesame 0xb404363 thru 0xb408474 data_size 0x1(0x4112 bytes) 2024-09-11T05:51:58.805 INFO:teuthology.orchestra.run.smithi165.stdout:3708 writesame 0x742c99d thru 0x742ffdf data_size 0x1(0x3643 bytes) 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:92946 3.c 3:32eb61c4:::rbd_data.112450dbd059.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:58.809452+0000", 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0061196080000000003, 2024-09-11T05:51:58.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061440990000000001, 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 92946 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809452+0000", 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809452+0000", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809454+0000", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0360000000000001e-06 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809463+0000", 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4010000000000003e-06 2024-09-11T05:51:58.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809482+0000", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9190000000000001e-05 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809493+0000", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0902e-05 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809556+0000", 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2377000000000003e-05 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.809624+0000", 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8272e-05 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:58.811238+0000", 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001614051 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-11T05:51:58.818 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:58.819 INFO:teuthology.orchestra.run.smithi165.stdout:3709 punch from 0x35db6d4 to 0x35e7279, (0xbba5 bytes) 2024-09-11T05:51:58.819 INFO:teuthology.orchestra.run.smithi165.stdout:3710 writesame 0x8ebe206 thru 0x8ec7520 data_size 0x1(0x931b bytes) 2024-09-11T05:51:58.830 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:58.828+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:58.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:51:58.854 INFO:teuthology.orchestra.run.smithi165.stdout:3712 write 0xc6e3f9a thru 0xc6eb2b1 (0x7318 bytes) 2024-09-11T05:51:58.872 INFO:teuthology.orchestra.run.smithi165.stdout:3713 trunc from 0xc6eb2b2 to 0x757c2ca 2024-09-11T05:51:58.893 INFO:teuthology.orchestra.run.smithi165.stdout:3714 write 0x321a344 thru 0x321fdda (0x5a97 bytes) 2024-09-11T05:51:58.897 INFO:teuthology.orchestra.run.smithi165.stdout:3715 read 0x629f49e thru 0x62a091c (0x147f bytes) 2024-09-11T05:51:58.898 INFO:teuthology.orchestra.run.smithi165.stdout:3716 write 0x708fb87 thru 0x7096d8b (0x7205 bytes) 2024-09-11T05:51:58.901 INFO:teuthology.orchestra.run.smithi165.stdout:3719 writesame 0x63e0f44 thru 0x63e4f99 data_size 0x5a(0x4056 bytes) 2024-09-11T05:51:58.919 INFO:teuthology.orchestra.run.smithi165.stdout:3720 trunc from 0x757c2ca to 0xdbeeea4 2024-09-11T05:51:58.931 INFO:teuthology.orchestra.run.smithi165.stdout:3721 write 0x8c7786 thru 0x8cb17a (0x39f5 bytes) 2024-09-11T05:51:58.931 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:58.928+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:58.934 INFO:teuthology.orchestra.run.smithi165.stdout:3723 writesame 0x98f2caf thru 0x98f7c4a data_size 0xa(0x4f9c bytes) 2024-09-11T05:51:58.942 INFO:teuthology.orchestra.run.smithi165.stdout:3727 write 0x7c8034c thru 0x7c8291f (0x25d4 bytes) 2024-09-11T05:51:58.946 INFO:teuthology.orchestra.run.smithi165.stdout:3728 write 0x9cd2878 thru 0x9cd5153 (0x28dc bytes) 2024-09-11T05:51:58.952 INFO:teuthology.orchestra.run.smithi165.stdout:3729 read 0x31ee365 thru 0x31f4891 (0x652d bytes) 2024-09-11T05:51:58.952 INFO:teuthology.orchestra.run.smithi165.stdout:3731 write 0x2fb1610 thru 0x2fbf478 (0xde69 bytes) 2024-09-11T05:51:58.954 INFO:teuthology.orchestra.run.smithi165.stdout:3732 writesame 0x882183f thru 0x8829636 data_size 0x1d(0x7df8 bytes) 2024-09-11T05:51:58.958 INFO:teuthology.orchestra.run.smithi165.stdout:3734 writesame 0x7fe6a4d thru 0x7feb609 data_size 0x17(0x4bbd bytes) 2024-09-11T05:51:58.961 INFO:teuthology.orchestra.run.smithi165.stdout:3736 read 0x762290 thru 0x7718e3 (0xf654 bytes) 2024-09-11T05:51:58.985 INFO:teuthology.orchestra.run.smithi165.stdout:3737 trunc from 0xdbeeea4 to 0x49c0845 2024-09-11T05:51:59.006 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.006 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:59.006 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:59.006 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:59.006 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:51:59.033 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:59.032+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:59.033 INFO:teuthology.orchestra.run.smithi165.stdout:3738 writesame 0x8a2b47d thru 0x8a31620 data_size 0x6(0x61a4 bytes) 2024-09-11T05:51:59.033 INFO:teuthology.orchestra.run.smithi165.stdout:3739 writesame 0xe7f84e thru 0xe8d91c data_size 0x13(0xe0cf bytes) 2024-09-11T05:51:59.038 INFO:teuthology.orchestra.run.smithi165.stdout:3740 clone 23 order 23 su 65536 sc 4 2024-09-11T05:51:59.133 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:59.132+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:59.138 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.378685922999999, 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:51:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:51:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.335457810000001, 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:51:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541896573999999, 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:51:59.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:51:59.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:51:59.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:51:59.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:59.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541734082000001, 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:51:59.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:51:59.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.158 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541709011000002, 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:59.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541684933999999, 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541634251000001, 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.167 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541604651, 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.171 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541579030000001, 2024-09-11T05:51:59.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:51:59.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541553651000001, 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:51:59.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:51:59.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:51:59.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541526140999999, 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:51:59.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:51:59.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:59.188 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541497024000002, 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:59.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:51:59.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541470760999999, 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541276302, 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541253179999998, 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:51:59.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:51:59.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:51:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541226202000001, 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:51:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:51:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:51:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541200443000001, 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:51:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541173351000001, 2024-09-11T05:51:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:51:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.541149507, 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:51:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:86155 3.1d 3:bed54d18:::rbd_data.11244627d646.000000000000028f:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.7804974470000001, 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043742621000000002, 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86155 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355884+0000", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3340000000000001e-06 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355890+0000", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7679999999999997e-06 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355903+0000", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.259e-05 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355909+0000", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9349999999999999e-06 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356023+0000", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114307 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356362+0000", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033904300000000003 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.365362+0000", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089999659999999999 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:51:59.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399542+0000", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034180413999999999 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399557+0000", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4681999999999999e-05 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399594+0000", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.646e-05 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399626+0000", 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2122000000000002e-05 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.227 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:51:59.277 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:51:59.275+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:51:59.335 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:59.332+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:59.352 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:51:59.356 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.357 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:59.357 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:59.357 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:51:59.458 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:51:59.455+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:51:59.477 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone22 from 0x8a31621 (overlap 0xaeb537) to 0x0 2024-09-11T05:51:59.537 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:51:59.536+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:51:59.539 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.539 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:59.539 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:51:59.539 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:51:59.539 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:51:59.608 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-09-11T05:51:59.638 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:51:59.636+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:1026 3.a e30/28 3:50a9d2c1:::rbd_data.112475afb12f.000000000000004c:head v 30'2, mlcod=0'0)", 2024-09-11T05:51:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.870446348999998, 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043959489999999997, 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851061+0000", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851063+0000", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899999999999999e-06 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851141+0000", 2024-09-11T05:51:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851182+0000", 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1207e-05 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851204+0000", 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1812999999999999e-05 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851341+0000", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013705600000000001 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.851391+0000", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9811999999999997e-05 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855349+0000", 2024-09-11T05:51:59.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00395825 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:12.855456+0000", 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107582 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:6402 3.a e33/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 33'21, mlcod=33'19)", 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.840240577000003, 2024-09-11T05:51:59.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003674802, 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881266+0000", 2024-09-11T05:51:59.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881269+0000", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6539999999999999e-06 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881325+0000", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609000000000002e-05 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881385+0000", 2024-09-11T05:51:59.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0043000000000001e-05 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881406+0000", 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1670000000000001e-05 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881582+0000", 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017577200000000001 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.881649+0000", 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6608999999999999e-05 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884821+0000", 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003172336 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:16.884941+0000", 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120109 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.728245364999999, 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:59.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:51:59.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7161 3.1e e34/23 3:7800550e:::rbd_data.11249b359231.0000000000000023:head v 34'27, mlcod=34'25)", 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.649071868999997, 2024-09-11T05:51:59.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044832179999999998, 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072435+0000", 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072438+0000", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1599999999999998e-06 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072465+0000", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6855999999999998e-05 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072505+0000", 2024-09-11T05:51:59.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0323e-05 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072527+0000", 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2016000000000001e-05 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072689+0000", 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161943 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.072751+0000", 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1241999999999994e-05 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076795+0000", 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040443759999999997 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:18.076918+0000", 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012330199999999999 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:11928 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'62, mlcod=36'60)", 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.116661508999996, 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037441639999999999, 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604845+0000", 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604846+0000", 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1230000000000001e-06 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604861+0000", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4877e-05 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604892+0000", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0973000000000003e-05 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.604915+0000", 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2796999999999999e-05 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605079+0000", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016418800000000001 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.605138+0000", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8579999999999998e-05 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608482+0000", 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033439770000000001 2024-09-11T05:51:59.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:21.608590+0000", 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107649 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:12634 3.a e37/28 3:56e5e968:::rbd_header.1124518b2c12:head v 37'87, mlcod=37'85)", 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.201200581999998, 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035953420000000001, 2024-09-11T05:51:59.746 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520306+0000", 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520308+0000", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8339999999999999e-06 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520321+0000", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2523e-05 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520351+0000", 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0580000000000002e-05 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520366+0000", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4866e-05 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520499+0000", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132636 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.520540+0000", 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0883000000000001e-05 2024-09-11T05:51:59.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523805+0000", 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032651749999999999 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.523902+0000", 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6844999999999999e-05 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:51:59.750 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.338528879999998, 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.517677198000001, 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:51:59.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:51:59.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.218498140000001, 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:85092 3.0 e60/56 3:06107e3f:::rbd_data.1124fc73d6eb.0000000000000003:head v 60'205, mlcod=60'203)", 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.158092699000001, 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038279239999999999, 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563414+0000", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.563416+0000", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8589999999999999e-06 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564065+0000", 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064892599999999997 2024-09-11T05:51:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564091+0000", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6299e-05 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564103+0000", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1914e-05 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564179+0000", 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5924e-05 2024-09-11T05:51:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.564202+0000", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2744000000000001e-05 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567191+0000", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029887300000000002 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:49.567242+0000", 2024-09-11T05:51:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1527999999999999e-05 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:89803 3.1e e64/23 3:79bfa39c:::rbd_data.11242d2cfbf4.0000000000000090:head v 64'199, mlcod=62'197)", 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0308822839999996, 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035203410000000002, 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690625+0000", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690627+0000", 2024-09-11T05:51:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690646+0000", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9173000000000001e-05 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690686+0000", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9541e-05 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.690707+0000", 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0988999999999998e-05 2024-09-11T05:51:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.691159+0000", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000452201 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.691210+0000", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1048000000000003e-05 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.694037+0000", 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002826402 2024-09-11T05:51:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:54.694145+0000", 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010836399999999999 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90436 3.1e e64/23 3:7990e080:::rbd_data.11242d2cfbf4.0000000000000086:head v 64'202, mlcod=64'200)", 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.7180944399999998, 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035215329999999999, 2024-09-11T05:51:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003412+0000", 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003414+0000", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6559999999999999e-06 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003507+0000", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3068999999999999e-05 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003525+0000", 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7363999999999998e-05 2024-09-11T05:51:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003534+0000", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0240000000000003e-06 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003607+0000", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3449999999999996e-05 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.003628+0000", 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509e-05 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.006878+0000", 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032491740000000001 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.006934+0000", 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6286999999999998e-05 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90695 3.0 e64/56 3:03391bbf:::rbd_data.11242d2cfbf4.0000000000000079:head v 64'248, mlcod=64'246)", 2024-09-11T05:51:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.466236+0000", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.2552712599999998, 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003742928, 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466236+0000", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466236+0000", 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466236+0000", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8199999999999999e-07 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466244+0000", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3350000000000001e-06 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466256+0000", 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2112e-05 2024-09-11T05:51:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466263+0000", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.832e-06 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466381+0000", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011843799999999999 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.466399+0000", 2024-09-11T05:51:59.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8077999999999999e-05 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.469934+0000", 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035344199999999999 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.469979+0000", 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4931000000000001e-05 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90878 3.1e e64/23 3:7d08a1c9:::rbd_data.11242d2cfbf4.000000000000007c:head v 64'209, mlcod=64'207)", 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.0426728089999999, 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053596290000000003, 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678835+0000", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4199999999999995e-07 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678840+0000", 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1270000000000002e-06 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678847+0000", 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-06 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678851+0000", 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5889999999999996e-06 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678918+0000", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6451000000000005e-05 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678960+0000", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2265000000000003e-05 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684084+0000", 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051238810000000003 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684194+0000", 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109744 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90881 3.a e64/28 3:538a4bad:::rbd_data.11242d2cfbf4.00000000000000bc:head v 64'253, mlcod=64'250)", 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.0411177609999998, 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068461709999999999, 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680390+0000", 2024-09-11T05:51:59.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6600000000000005e-07 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680396+0000", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3940000000000001e-06 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680412+0000", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5208000000000001e-05 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680421+0000", 2024-09-11T05:51:59.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.628e-06 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680497+0000", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5315000000000007e-05 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680517+0000", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0437000000000001e-05 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687105+0000", 2024-09-11T05:51:59.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00658787 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687235+0000", 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013045299999999999 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90886 3.1e e64/23 3:7a4cea0f:::rbd_data.11242d2cfbf4.00000000000001b5:head v 64'210, mlcod=64'207)", 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.0392706799999996, 2024-09-11T05:51:59.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004920804, 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682237+0000", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9199999999999999e-07 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682243+0000", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2920000000000001e-06 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682258+0000", 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4062e-05 2024-09-11T05:51:59.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682264+0000", 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9059999999999996e-06 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682322+0000", 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7989000000000003e-05 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682343+0000", 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0307999999999999e-05 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687116+0000", 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047728049999999998 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687157+0000", 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1449999999999998e-05 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:91050 3.a e64/28 3:545a6621:::rbd_data.11242d2cfbf4.0000000000000055:head v 64'256, mlcod=64'254)", 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:56.049852+0000", 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6716546839999999, 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003662532, 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049852+0000", 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049852+0000", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049855+0000", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5160000000000001e-06 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049886+0000", 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1381999999999999e-05 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049918+0000", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1825000000000001e-05 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.049934+0000", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5815999999999999e-05 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.050030+0000", 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5869e-05 2024-09-11T05:51:59.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.050066+0000", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6387000000000002e-05 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.053463+0000", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033969360000000001 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.053515+0000", 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1801000000000001e-05 2024-09-11T05:51:59.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:91384 3.0 e64/56 3:0507e7ab:::rbd_data.11242d2cfbf4.0000000000000007:head v 64'253, mlcod=64'251)", 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:56.119997+0000", 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6015099930000001, 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035048200000000001, 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.119997+0000", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.119997+0000", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.119998+0000", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8599999999999997e-07 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.120627+0000", 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062907900000000005 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.120640+0000", 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3165e-05 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.120648+0000", 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3720000000000005e-06 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.120738+0000", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9998000000000001e-05 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.120780+0000", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1705999999999997e-05 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.123389+0000", 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026086630000000002 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.123502+0000", 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112951 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:91434 3.1e e64/23 3:7835f8eb:::rbd_data.11242d2cfbf4.000000000000011c:head v 64'213, mlcod=64'211)", 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.468294099, 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047778619999999999, 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253216+0000", 2024-09-11T05:51:59.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.117e-06 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253326+0000", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110404 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253370+0000", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3927999999999997e-05 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253393+0000", 2024-09-11T05:51:59.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2758000000000001e-05 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253552+0000", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159092 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253617+0000", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5170000000000001e-05 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257880+0000", 2024-09-11T05:51:59.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042626649999999997 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257991+0000", 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110728 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:93118 3.a e67/28 3:53ac2b01:::rbd_data.112450dbd059.0000000000000000:head v 67'287, mlcod=66'284)", 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:59.483659+0000", 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.23784765099999999, 2024-09-11T05:51:59.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045625650000000002, 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483659+0000", 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483659+0000", 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483660+0000", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.035e-06 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483669+0000", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9809999999999992e-06 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483685+0000", 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5330999999999999e-05 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483693+0000", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5339999999999999e-06 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483814+0000", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012090400000000001 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.483836+0000", 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1747999999999999e-05 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.488165+0000", 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043292060000000004 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:59.488222+0000", 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6826000000000001e-05 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:51:59.810 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.811 INFO:teuthology.orchestra.run.smithi165.stdout:3741 trunc from 0x8a31621 to 0x28c09ef 2024-09-11T05:51:59.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:51:59.812 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:51:59.812+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:51:59.815 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:51:59.815 INFO:teuthology.orchestra.run.smithi165.stdout:3745 punch from 0x109ca09 to 0x10a228a, (0x5881 bytes) 2024-09-11T05:51:59.849 INFO:teuthology.orchestra.run.smithi165.stdout:3746 writesame 0xcdb54c1 thru 0xcdc3b7f data_size 0x1(0xe6bf bytes) 2024-09-11T05:51:59.867 INFO:teuthology.orchestra.run.smithi165.stdout:3747 trunc from 0xcdc3b80 to 0x489564f 2024-09-11T05:51:59.909 INFO:teuthology.orchestra.run.smithi165.stdout:3748 write 0xd3f3d82 thru 0xd3fc204 (0x8483 bytes) 2024-09-11T05:51:59.912 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:51:59.912+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:51:59.916 INFO:teuthology.orchestra.run.smithi165.stdout:3750 clone 24 order 20 su 65536 sc 15 2024-09-11T05:51:59.937 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:51:59.937 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:51:59.937 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:51:59.937 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:51:59.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:00.013 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:00.012+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:00.073 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:00.073 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:00.073 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:00.073 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:00.073 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:00.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:00.114 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:00.112+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.102263291, 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:52:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:52:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:52:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.860675908000001, 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.860065081999998, 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:52:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.860040675, 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.860030356999999, 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857426742000001, 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:52:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857416300000001, 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857404592999998, 2024-09-11T05:52:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857393513000002, 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:52:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:52:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857358418, 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857317911999999, 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:52:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857284115999999, 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:00.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:52:00.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857245212999999, 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.265 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:52:00.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857158712, 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:52:00.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:52:00.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.857141451, 2024-09-11T05:52:00.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:52:00.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.856835376999999, 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:52:00.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:52:00.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.856775603999999, 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:52:00.283 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:52:00.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.856753960999999, 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.856732758, 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.291 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.855687967000001, 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:00.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:00.301 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:00.300+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:00.401 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:00.395+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:00.441 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:00.441 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:00.441 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:00.441 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:00.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:00.492 INFO:teuthology.orchestra.run.smithi165.stdout:flattening image image_client.0-clone23 2024-09-11T05:52:00.502 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:00.499+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:00.598 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:00.598 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:00.598 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:00.598 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:00.598 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:00.603 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:00.599+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:00.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:00.712 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:00.712+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:00.771 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-09-11T05:52:00.775 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:00.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.983469403, 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:00.787 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:00.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:00.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9834424239999997, 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:00.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:00.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.797 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9834121649999998, 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:00.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:00.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:00.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9833668800000002, 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.803 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:00.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.807 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9833340140000004, 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.808 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.983305959, 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.813 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:00.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:00.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:00.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:00.818 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9832788929999996, 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:00.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9832288919999996, 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.824 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:00.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:00.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9831964089999996, 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:00.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9831679399999995, 2024-09-11T05:52:00.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:00.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:00.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:00.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.839 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9831390610000001, 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:00.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9830822430000001, 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:00.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:00.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9830512110000003, 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.850 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:00.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:00.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:00.855 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9830231920000001, 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:00.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:00.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9829962630000004, 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.861 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:00.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9829696969999997, 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.866 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:00.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9829324860000002, 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.871 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:00.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9829039330000002, 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.876 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:00.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9828554719999998, 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:00.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9828168569999995, 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:00.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:00.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:00.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:00.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:00.891 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:00.892 INFO:teuthology.orchestra.run.smithi165.stdout:3752 write 0xd9ffba5 thru 0xda0a457 (0xa8b3 bytes) 2024-09-11T05:52:00.892 INFO:teuthology.orchestra.run.smithi165.stdout:3753 writesame 0xca56360 thru 0xca6191a data_size 0x1(0xb5bb bytes) 2024-09-11T05:52:00.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3754 write 0x522b8f3 thru 0x5236465 (0xab73 bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3755 read 0x252e04e thru 0x253a3dd (0xc390 bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3757 read 0xa3114da thru 0xa3187d0 (0x72f7 bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3758 read 0x311f711 thru 0x312c4bd (0xcdad bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3759 read 0x125cfa6 thru 0x1269fe1 (0xd03c bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3761 read 0x78da8f1 thru 0x78dcb1e (0x222e bytes) 2024-09-11T05:52:00.894 INFO:teuthology.orchestra.run.smithi165.stdout:3762 punch from 0x419b997 to 0x41a1f34, (0x659d bytes) 2024-09-11T05:52:00.895 INFO:teuthology.orchestra.run.smithi165.stdout:3764 writesame 0x1d5e31f thru 0x1d6785b data_size 0x2d(0x953d bytes) 2024-09-11T05:52:00.895 INFO:teuthology.orchestra.run.smithi165.stdout:3765 punch from 0x74c5700 to 0x74ce569, (0x8e69 bytes) 2024-09-11T05:52:00.895 INFO:teuthology.orchestra.run.smithi165.stdout:3766 trunc from 0xda0a458 to 0x5a1829c 2024-09-11T05:52:00.895 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:00.892+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:00.918 INFO:teuthology.orchestra.run.smithi165.stdout:3767 trunc from 0x5a1829c to 0x144d478 2024-09-11T05:52:00.944 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:00.944 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:00.964 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:00.964 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:00.982 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:00.983 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:00.993 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:00.992+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:01.006 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:01.006 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:01.010 INFO:teuthology.orchestra.run.smithi165.stdout:3769 write 0x6f57a76 thru 0x6f5f184 (0x770f bytes) 2024-09-11T05:52:01.016 INFO:teuthology.orchestra.run.smithi165.stdout:3770 read 0x56e28f4 thru 0x56e9bab (0x72b8 bytes) 2024-09-11T05:52:01.016 INFO:teuthology.orchestra.run.smithi165.stdout:3771 read 0x332d81c thru 0x333b67d (0xde62 bytes) 2024-09-11T05:52:01.016 INFO:teuthology.orchestra.run.smithi165.stdout:3775 write 0x32f1bad thru 0x32f47c1 (0x2c15 bytes) 2024-09-11T05:52:01.033 INFO:teuthology.orchestra.run.smithi165.stdout:3776 write 0x943e283 thru 0x943fca3 (0x1a21 bytes) 2024-09-11T05:52:01.039 INFO:teuthology.orchestra.run.smithi165.stdout:3778 read 0x3e22e38 thru 0x3e2a11e (0x72e7 bytes) 2024-09-11T05:52:01.039 INFO:teuthology.orchestra.run.smithi165.stdout:3780 clone 25 order 18 su 1024 sc 10 2024-09-11T05:52:01.041 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:01.041 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:01.068 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.068 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.073 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:01.073 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:01.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:01.094 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:01.091+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:01.095 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:01.095 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:01.115 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:01.115 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:01.195 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:01.192+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:01.209 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.209 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:01.296 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:01.291+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:01.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:52:01.397 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:01.395+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:01.400 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.400 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:01.469 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone24 from 0x943fca4 (overlap 0x144d478) to 0x40f74b 2024-09-11T05:52:01.498 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:01.496+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:01.594 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.594 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.599 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:01.596+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:01.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:01.633 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-09-11T05:52:01.700 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:01.699+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:01.757 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.757 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:01.820 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:01.820+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:01.895 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:01.895 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:01.901 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:01.900+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:01.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:02.003 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:01.999+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:02.010 INFO:teuthology.orchestra.run.smithi165.stdout:3781 read 0x398b74a thru 0x399618c (0xaa43 bytes) 2024-09-11T05:52:02.013 INFO:teuthology.orchestra.run.smithi165.stdout:3782 write 0x8104af7 thru 0x8108fee (0x44f8 bytes) 2024-09-11T05:52:02.036 INFO:teuthology.orchestra.run.smithi165.stdout:3783 write 0x20a3a5 thru 0x217a63 (0xd6bf bytes) 2024-09-11T05:52:02.104 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:02.104+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:02.113 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:02.113 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:02.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:02.220 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:02.220+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:02.238 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:3785 read 0x91ef96d thru 0x91f4ab3 (0x5147 bytes) 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:3786 read 0x295468d thru 0x2957c14 (0x3588 bytes) 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:3788 read 0x3619c3e thru 0x362664b (0xca0e bytes) 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:3790 punch from 0x1f9889e to 0x1f9ae6c, (0x25ce bytes) 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:02.239 INFO:teuthology.orchestra.run.smithi165.stdout:3792 writesame 0xed9034 thru 0xee2feb data_size 0x26(0x9fb8 bytes) 2024-09-11T05:52:02.305 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:02.304+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:02.316 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:02.316 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:02.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:02.368 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:52:02.389 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:02.389 INFO:teuthology.orchestra.run.smithi165.stdout:3794 punch from 0x6db8b0a to 0x6dc6c1d, (0xe113 bytes) 2024-09-11T05:52:02.391 INFO:teuthology.orchestra.run.smithi165.stdout:3795 trunc from 0x943fca4 to 0x790ed1b 2024-09-11T05:52:02.406 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:02.403+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:02.433 INFO:teuthology.orchestra.run.smithi165.stdout:3796 write 0xad3ab9d thru 0xad47d4d (0xd1b1 bytes) 2024-09-11T05:52:02.458 INFO:teuthology.orchestra.run.smithi165.stdout:3797 write 0xe6c5bff thru 0xe6c880c (0x2c0e bytes) 2024-09-11T05:52:02.474 INFO:teuthology.orchestra.run.smithi165.stdout:3800 read 0x143a04d thru 0x143e782 (0x4736 bytes) 2024-09-11T05:52:02.477 INFO:teuthology.orchestra.run.smithi165.stdout:3802 write 0x8d610d5 thru 0x8d70eef (0xfe1b bytes) 2024-09-11T05:52:02.490 INFO:teuthology.orchestra.run.smithi165.stdout:3804 read 0x929eb62 thru 0x92a73a5 (0x8844 bytes) 2024-09-11T05:52:02.493 INFO:teuthology.orchestra.run.smithi165.stdout:3805 trunc from 0xe6c880d to 0xc7978ce 2024-09-11T05:52:02.499 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:02.499 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:02.507 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:02.503+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:02.513 INFO:teuthology.orchestra.run.smithi165.stdout:3806 write 0x5f4f79f thru 0x5f5acee (0xb550 bytes) 2024-09-11T05:52:02.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:02.532 INFO:teuthology.orchestra.run.smithi165.stdout:3807 read 0x2fd67b7 thru 0x2fdfb5f (0x93a9 bytes) 2024-09-11T05:52:02.533 INFO:teuthology.orchestra.run.smithi165.stdout:3808 writesame 0xbcfa11c thru 0xbcfd599 data_size 0x1(0x347e bytes) 2024-09-11T05:52:02.549 INFO:teuthology.orchestra.run.smithi165.stdout:3809 writesame 0x5049854 thru 0x5055c36 data_size 0x1(0xc3e3 bytes) 2024-09-11T05:52:02.568 INFO:teuthology.orchestra.run.smithi165.stdout:3810 writesame 0x189546c thru 0x18a0853 data_size 0x39(0xb3e8 bytes) 2024-09-11T05:52:02.609 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:02.603+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:02.663 INFO:teuthology.orchestra.run.smithi165.stdout:3811 read 0xc3af7b8 thru 0xc3b7171 (0x79ba bytes) 2024-09-11T05:52:02.673 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:02.673 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:02.675 INFO:teuthology.orchestra.run.smithi165.stdout:3812 trunc from 0xc7978ce to 0x355834f 2024-09-11T05:52:02.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:02.709 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:02.708+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:02.752 INFO:teuthology.orchestra.run.smithi165.stdout:3813 write 0x7f50a79 thru 0x7f5994e (0x8ed6 bytes) 2024-09-11T05:52:02.779 INFO:teuthology.orchestra.run.smithi165.stdout:3814 write 0xb891f00 thru 0xb89834a (0x644b bytes) 2024-09-11T05:52:02.795 INFO:teuthology.orchestra.run.smithi165.stdout:3816 read 0x1fabb45 thru 0x1fb580d (0x9cc9 bytes) 2024-09-11T05:52:02.795 INFO:teuthology.orchestra.run.smithi165.stdout:3818 read 0x9433e72 thru 0x94381ba (0x4349 bytes) 2024-09-11T05:52:02.803 INFO:teuthology.orchestra.run.smithi165.stdout:3819 write 0xcf9854e thru 0xcf9990e (0x13c1 bytes) 2024-09-11T05:52:02.810 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:02.808+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:02.817 INFO:teuthology.orchestra.run.smithi165.stdout:3820 read 0x364700a thru 0x36556a8 (0xe69f bytes) 2024-09-11T05:52:02.818 INFO:teuthology.orchestra.run.smithi165.stdout:3821 write 0x3ee967b thru 0x3eeec0c (0x5592 bytes) 2024-09-11T05:52:02.833 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:02.833 INFO:teuthology.orchestra.run.smithi165.stdout:3823 punch from 0xa1c38dc to 0xa1ced92, (0xb4b6 bytes) 2024-09-11T05:52:02.848 INFO:teuthology.orchestra.run.smithi165.stdout:3824 trunc from 0xcf9990f to 0x5517238 2024-09-11T05:52:02.871 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:02.872 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:02.872 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:02.884 INFO:teuthology.orchestra.run.smithi165.stdout:3827 write 0x5f2b633 thru 0x5f37604 (0xbfd2 bytes) 2024-09-11T05:52:02.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:02.919 INFO:teuthology.orchestra.run.smithi165.stdout:3828 write 0x9ccd83d thru 0x9ccdfd8 (0x79c bytes) 2024-09-11T05:52:02.928 INFO:teuthology.orchestra.run.smithi165.stdout:3831 punch from 0x8f56824 to 0x8f5ee49, (0x8625 bytes) 2024-09-11T05:52:02.928 INFO:teuthology.orchestra.run.smithi165.stdout:3833 punch from 0x3d60099 to 0x3d67703, (0x766a bytes) 2024-09-11T05:52:02.928 INFO:teuthology.orchestra.run.smithi165.stdout:3834 punch from 0x858b169 to 0x858c709, (0x15a0 bytes) 2024-09-11T05:52:02.928 INFO:teuthology.orchestra.run.smithi165.stdout:3835 read 0x4c79ef6 thru 0x4c810a8 (0x71b3 bytes) 2024-09-11T05:52:02.929 INFO:teuthology.orchestra.run.smithi165.stdout:3836 write 0x24f9ab0 thru 0x25019dd (0x7f2e bytes) 2024-09-11T05:52:02.929 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:02.927+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:02.974 INFO:teuthology.orchestra.run.smithi165.stdout:3839 write 0xc110d3e thru 0xc1161e2 (0x54a5 bytes) 2024-09-11T05:52:02.999 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.011 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:03.008+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:03.019 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:03.019 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:03.023 INFO:teuthology.orchestra.run.smithi165.stdout:3841 trunc from 0xc1161e3 to 0x1c68c2c 2024-09-11T05:52:03.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:03.044 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:03.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T05:52:03.112 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:03.112+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:03.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:03.193 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.193 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.213 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:03.212+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:03.213 INFO:teuthology.orchestra.run.smithi165.stdout:3845 write 0x914cba5 thru 0x9151cfc (0x5158 bytes) 2024-09-11T05:52:03.227 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:03.227 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:03.227 INFO:teuthology.orchestra.run.smithi165.stdout:3846 read 0x119c1a8 thru 0x11a2f9b (0x6df4 bytes) 2024-09-11T05:52:03.231 INFO:teuthology.orchestra.run.smithi165.stdout:3847 writesame 0x2a446f5 thru 0x2a4ec3f data_size 0x69(0xa54b bytes) 2024-09-11T05:52:03.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:03.248 INFO:teuthology.orchestra.run.smithi165.stdout:3848 punch from 0x25049e3 to 0x251476f, (0xfd8c bytes) 2024-09-11T05:52:03.248 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.248 INFO:teuthology.orchestra.run.smithi165.stdout:3852 punch from 0x57e5148 to 0x57e8bee, (0x3aa6 bytes) 2024-09-11T05:52:03.248 INFO:teuthology.orchestra.run.smithi165.stdout:3853 read 0x768a0f thru 0x773948 (0xaf3a bytes) 2024-09-11T05:52:03.269 INFO:teuthology.orchestra.run.smithi165.stdout:3854 trunc from 0x9151cfd to 0x233b525 2024-09-11T05:52:03.290 INFO:teuthology.orchestra.run.smithi165.stdout:3857 trunc from 0x233b525 to 0x71e32f7 2024-09-11T05:52:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:3858 read 0x66cb0cf thru 0x66d18ce (0x6800 bytes) 2024-09-11T05:52:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:3860 read 0x3c51636 thru 0x3c60138 (0xeb03 bytes) 2024-09-11T05:52:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:3861 read 0x4933a3b thru 0x4937054 (0x361a bytes) 2024-09-11T05:52:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:3862 read 0xd81bfd thru 0xd85d50 (0x4154 bytes) 2024-09-11T05:52:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:3863 read 0x56999e0 thru 0x569f9f8 (0x6019 bytes) 2024-09-11T05:52:03.299 INFO:teuthology.orchestra.run.smithi165.stdout:3864 read 0xa8127e thru 0xa85f8f (0x4d12 bytes) 2024-09-11T05:52:03.303 INFO:teuthology.orchestra.run.smithi165.stdout:3865 trunc from 0x71e32f7 to 0x8855d3f 2024-09-11T05:52:03.309 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.309 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.314 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:03.312+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:03.318 INFO:teuthology.orchestra.run.smithi165.stdout:3870 trunc from 0x8855d3f to 0x25d87a0 2024-09-11T05:52:03.319 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:03.319 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:03.325 INFO:teuthology.orchestra.run.smithi165.stdout:3872 read 0x1adbccf thru 0x1ade65f (0x2991 bytes) 2024-09-11T05:52:03.339 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:03.340 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:03.340 INFO:teuthology.orchestra.run.smithi165.stdout:3873 trunc from 0x25d87a0 to 0xad5f730 2024-09-11T05:52:03.347 INFO:teuthology.orchestra.run.smithi165.stdout:3874 write 0x76f36e3 thru 0x76f45b0 (0xece bytes) 2024-09-11T05:52:03.357 INFO:teuthology.orchestra.run.smithi165.stdout:3876 writesame 0x4ba467b thru 0x4bb349f data_size 0x5(0xee25 bytes) 2024-09-11T05:52:03.361 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:03.361 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:03.372 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:03.372 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:03.377 INFO:teuthology.orchestra.run.smithi165.stdout:3878 read 0x33e2698 thru 0x33e6cfa (0x4663 bytes) 2024-09-11T05:52:03.378 INFO:teuthology.orchestra.run.smithi165.stdout:3879 writesame 0x41894bf thru 0x4195531 data_size 0x1(0xc073 bytes) 2024-09-11T05:52:03.385 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:03.385 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:03.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:03.402 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.405 INFO:teuthology.orchestra.run.smithi165.stdout:3881 trunc from 0xad5f730 to 0x876850a 2024-09-11T05:52:03.406 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:03.406 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:03.412 INFO:teuthology.orchestra.run.smithi165.stdout:3882 write 0x3e4b9bd thru 0x3e4f4c9 (0x3b0d bytes) 2024-09-11T05:52:03.428 INFO:teuthology.orchestra.run.smithi165.stdout:3883 writesame 0x2252d06 thru 0x2260d79 data_size 0x34(0xe074 bytes) 2024-09-11T05:52:03.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:03.430 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:03.433 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:03.431+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:03.448 INFO:teuthology.orchestra.run.smithi165.stdout:3884 trunc from 0x876850a to 0x543b33e 2024-09-11T05:52:03.452 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:03.452 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:03.469 INFO:teuthology.orchestra.run.smithi165.stdout:3887 trunc from 0x543b33e to 0x98775f8 2024-09-11T05:52:03.474 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:03.474 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:3889 punch from 0x7d24e83 to 0x7d2d098, (0x8215 bytes) 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.477 INFO:teuthology.orchestra.run.smithi165.stdout:3895 writesame 0x1f67f6f thru 0x1f67ff1 data_size 0x1(0x83 bytes) 2024-09-11T05:52:03.480 INFO:teuthology.orchestra.run.smithi165.stdout:3896 punch from 0x193d195 to 0x19457d7, (0x8642 bytes) 2024-09-11T05:52:03.489 INFO:teuthology.orchestra.run.smithi165.stdout:3898 write 0xb22f1c9 thru 0xb239af8 (0xa930 bytes) 2024-09-11T05:52:03.506 INFO:teuthology.orchestra.run.smithi165.stdout:3899 writesame 0xa5101f6 thru 0xa517f92 data_size 0x1b(0x7d9d bytes) 2024-09-11T05:52:03.516 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:03.511+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:03.522 INFO:teuthology.orchestra.run.smithi165.stdout:3900 punch from 0x3c89a89 to 0x3c8e6d8, (0x4c4f bytes) 2024-09-11T05:52:03.522 INFO:teuthology.orchestra.run.smithi165.stdout:3901 write 0xacbfc2a thru 0xacc5ac9 (0x5ea0 bytes) 2024-09-11T05:52:03.526 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12728 3.1d 3:bec603fd:::rbd_data.1124518b2c12.0000000000000016:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.767938825000002, 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035544899999999997, 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.527 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12728 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757695+0000", 2024-09-11T05:52:03.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757698+0000", 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757711+0000", 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3713000000000001e-05 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757743+0000", 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1723000000000001e-05 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757758+0000", 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5017e-05 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.757877+0000", 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118329 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.758101+0000", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022457799999999999 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.764326+0000", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247659999999996 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793035+0000", 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028709515000000001 2024-09-11T05:52:03.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793070+0000", 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4826000000000002e-05 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793158+0000", 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7909000000000001e-05 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.793240+0000", 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1434000000000007e-05 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:26668 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 12077205] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.724710712, 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039583502999999999, 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:03.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26668 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800923+0000", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.800924+0000", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8700000000000004e-07 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808421+0000", 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976340000000004 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808443+0000", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.198e-05 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808454+0000", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0883000000000001e-05 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808565+0000", 2024-09-11T05:52:03.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110365 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.808697+0000", 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013242199999999999 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.809492+0000", 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079498100000000005 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:03.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.837677+0000", 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028184717000000001 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840432+0000", 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027555850000000001 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840470+0000", 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7725000000000002e-05 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:32.840506+0000", 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6223999999999997e-05 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68535 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.931149476000002, 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030620649, 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.539 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68535 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594484+0000", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594485+0000", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0800000000000005e-07 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594487+0000", 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.886e-06 2024-09-11T05:52:03.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594495+0000", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1319999999999996e-06 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594501+0000", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5309999999999998e-06 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.624934+0000", 2024-09-11T05:52:03.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030433427999999998 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625009+0000", 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4561999999999995e-05 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625105+0000", 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5601999999999997e-05 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68541 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2850816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930986984, 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030639631000000001, 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68541 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594647+0000", 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1699999999999998e-07 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594650+0000", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.869e-06 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594657+0000", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0570000000000004e-06 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594663+0000", 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:03.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625152+0000", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030489049000000001 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625219+0000", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6173999999999994e-05 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625286+0000", 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7625000000000004e-05 2024-09-11T05:52:03.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68542 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2916352~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930961913000001, 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030823968, 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.548 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68542 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594672+0000", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3300000000000003e-07 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594675+0000", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6759999999999999e-06 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594682+0000", 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3479999999999998e-06 2024-09-11T05:52:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594688+0000", 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9120000000000003e-06 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625329+0000", 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030641471 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625421+0000", 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1978000000000006e-05 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625496+0000", 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.415e-05 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68543 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 2981888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930937835999998, 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030971747000000001, 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68543 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594696+0000", 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594698+0000", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594705+0000", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594711+0000", 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8170000000000004e-06 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625539+0000", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030827394000000001 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625604+0000", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5031999999999997e-05 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625667+0000", 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3831999999999995e-05 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68544 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930887153, 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031106515000000001, 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68544 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594746+0000", 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594747+0000", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0210000000000001e-06 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594751+0000", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5020000000000001e-06 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594759+0000", 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8220000000000007e-06 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594768+0000", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8789999999999992e-06 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625710+0000", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030942067 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625780+0000", 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0623999999999993e-05 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625853+0000", 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000003e-05 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68545 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 3112960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930857552999999, 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246204, 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68545 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594776+0000", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594777+0000", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1500000000000004e-07 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594780+0000", 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2480000000000001e-06 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594787+0000", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1749999999999999e-06 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594793+0000", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3169999999999998e-06 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625894+0000", 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031101193999999999 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.625958+0000", 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3336999999999993e-05 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626022+0000", 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4318000000000003e-05 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68627 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930831932, 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.03143613, 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68627 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:52:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594802+0000", 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5400000000000001e-07 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594805+0000", 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7480000000000002e-06 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594812+0000", 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.255e-06 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594818+0000", 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2539999999999998e-06 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626065+0000", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031246494 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626138+0000", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3387999999999996e-05 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626238+0000", 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9437000000000004e-05 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68628 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930806553, 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031622011999999998, 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68628 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594827+0000", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594831+0000", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.484e-06 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594840+0000", 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7970000000000006e-06 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594845+0000", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5430000000000001e-06 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626284+0000", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031438541 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626361+0000", 2024-09-11T05:52:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7405999999999997e-05 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626449+0000", 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7688999999999993e-05 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68629 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930779043000001, 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812774000000002, 2024-09-11T05:52:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68629 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594854+0000", 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594855+0000", 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8400000000000004e-07 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594858+0000", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1140000000000002e-06 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594868+0000", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01e-05 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594875+0000", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626498+0000", 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031623198999999998 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626571+0000", 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3083999999999993e-05 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626667+0000", 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6094000000000005e-05 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68630 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930749926000001, 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031983769000000002, 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68630 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594884+0000", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594887+0000", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594893+0000", 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3620000000000002e-06 2024-09-11T05:52:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594901+0000", 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6210000000000004e-06 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626713+0000", 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031812566 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626784+0000", 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0510000000000001e-05 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626867+0000", 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3404000000000003e-05 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68631 3.16 3:683be82b:::rbd_data.1124a9e466a.0000000000000001:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930723662999998, 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032183798, 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68631 2024-09-11T05:52:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594910+0000", 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594911+0000", 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6200000000000006e-07 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594914+0000", 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594921+0000", 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3089999999999998e-06 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.594927+0000", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7910000000000001e-06 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626911+0000", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031984695 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.626984+0000", 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2143000000000004e-05 2024-09-11T05:52:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627094+0000", 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110049 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68637 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930529203999999, 2024-09-11T05:52:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032196904999999998, 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68637 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:52:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595104+0000", 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595105+0000", 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8299999999999997e-07 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595107+0000", 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5440000000000001e-06 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595114+0000", 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3509999999999998e-06 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595119+0000", 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1869999999999996e-06 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627143+0000", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032023972999999997 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627210+0000", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6940000000000006e-05 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627301+0000", 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1427000000000004e-05 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68638 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930506082000001, 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032386782000000003, 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68638 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595127+0000", 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595128+0000", 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9399999999999995e-07 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595131+0000", 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9239999999999999e-06 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595139+0000", 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0590000000000003e-06 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595145+0000", 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8150000000000002e-06 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627345+0000", 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032200564000000001 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627417+0000", 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2183999999999999e-05 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627514+0000", 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6742000000000003e-05 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68639 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930479104, 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032558616999999998, 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68639 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595154+0000", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595155+0000", 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595158+0000", 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9730000000000002e-06 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595166+0000", 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595172+0000", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627561+0000", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032389845 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627625+0000", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3742000000000001e-05 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627713+0000", 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7911999999999994e-05 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68640 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930453345, 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032748202999999997, 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68640 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595180+0000", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595181+0000", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595184+0000", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.067e-06 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595191+0000", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7419999999999997e-06 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595197+0000", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4060000000000004e-06 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627758+0000", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032560956000000002 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627833+0000", 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4770000000000004e-05 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627928+0000", 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5737999999999997e-05 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68641 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930426253, 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032922162999999997, 2024-09-11T05:52:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68641 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595207+0000", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595208+0000", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.99e-07 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595211+0000", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0350000000000002e-06 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595218+0000", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3339999999999996e-06 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595224+0000", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0589999999999999e-06 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.627975+0000", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032750964 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628041+0000", 2024-09-11T05:52:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5605000000000006e-05 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628129+0000", 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8566999999999994e-05 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:68642 3.16 3:68df2f87:::rbd_data.1124a9e466a.0000000000000005:10 [read 655360~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.930402408999999, 2024-09-11T05:52:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033103661999999999, 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 68642 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595231+0000", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1699999999999999e-07 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595234+0000", 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7240000000000001e-06 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595242+0000", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8509999999999999e-06 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.595248+0000", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1240000000000002e-06 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628163+0000", 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032914550000000001 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628239+0000", 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6464000000000005e-05 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.628335+0000", 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5532000000000006e-05 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:86155 3.1d 3:bed54d18:::rbd_data.11244627d646.000000000000028f:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T05:52:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.169750348999999, 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043742621000000002, 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86155 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355883+0000", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355884+0000", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3340000000000001e-06 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355890+0000", 2024-09-11T05:52:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7679999999999997e-06 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355903+0000", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.259e-05 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.355909+0000", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9349999999999999e-06 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356023+0000", 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114307 2024-09-11T05:52:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.356362+0000", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033904300000000003 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.365362+0000", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089999659999999999 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399542+0000", 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034180413999999999 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399557+0000", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4681999999999999e-05 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399594+0000", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.646e-05 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:51.399626+0000", 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2122000000000002e-05 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:03.620 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:03.621 INFO:teuthology.orchestra.run.smithi165.stdout:3902 write 0xaee35d0 thru 0xaef1bc9 (0xe5fa bytes) 2024-09-11T05:52:03.621 INFO:teuthology.orchestra.run.smithi165.stdout:3903 trunc from 0xb239af9 to 0x20daa49 2024-09-11T05:52:03.621 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.621 INFO:teuthology.orchestra.run.smithi165.stdout:3905 write 0x9df0c0a thru 0x9dfe5ec (0xd9e3 bytes) 2024-09-11T05:52:03.622 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:03.620+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:03.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:03.630 INFO:teuthology.orchestra.run.smithi165.stdout:3906 read 0x906103c thru 0x906aa36 (0x99fb bytes) 2024-09-11T05:52:03.646 INFO:teuthology.orchestra.run.smithi165.stdout:3907 writesame 0xe78b633 thru 0xe7955b4 data_size 0x11(0x9f82 bytes) 2024-09-11T05:52:03.646 INFO:teuthology.orchestra.run.smithi165.stdout:3908 read 0x9a9df65 thru 0x9aa4dde (0x6e7a bytes) 2024-09-11T05:52:03.647 INFO:teuthology.orchestra.run.smithi165.stdout:3911 write 0x10f0097 thru 0x10fd782 (0xd6ec bytes) 2024-09-11T05:52:03.724 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:03.724+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:03.783 INFO:teuthology.orchestra.run.smithi165.stdout:3912 write 0x82e0906 thru 0x82e6deb (0x64e6 bytes) 2024-09-11T05:52:03.795 INFO:teuthology.orchestra.run.smithi165.stdout:3913 write 0xce65f65 thru 0xce73f65 (0xe001 bytes) 2024-09-11T05:52:03.812 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:03.812 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:03.812 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:03.812 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:03.818 INFO:teuthology.orchestra.run.smithi165.stdout:3914 trunc from 0xe7955b5 to 0x835d0ea 2024-09-11T05:52:03.823 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:03.820+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:03.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:03.842 INFO:teuthology.orchestra.run.smithi165.stdout:3915 read 0x33ef2f2 thru 0x33f0999 (0x16a8 bytes) 2024-09-11T05:52:03.842 INFO:teuthology.orchestra.run.smithi165.stdout:3916 punch from 0x12be371 to 0x12c2105, (0x3d94 bytes) 2024-09-11T05:52:03.842 INFO:teuthology.orchestra.run.smithi165.stdout:3918 read 0x3e9c5a7 thru 0x3ea7a65 (0xb4bf bytes) 2024-09-11T05:52:03.843 INFO:teuthology.orchestra.run.smithi165.stdout:3922 read 0x61e17e1 thru 0x61ec9bb (0xb1db bytes) 2024-09-11T05:52:03.843 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:03.843 INFO:teuthology.orchestra.run.smithi165.stdout:3924 read 0x73df52a thru 0x73e5426 (0x5efd bytes) 2024-09-11T05:52:03.849 INFO:teuthology.orchestra.run.smithi165.stdout:3925 trunc from 0x835d0ea to 0x380bbdb 2024-09-11T05:52:03.863 INFO:teuthology.orchestra.run.smithi165.stdout:3926 punch from 0x2e2f540 to 0x2e3eb2a, (0xf5ea bytes) 2024-09-11T05:52:03.864 INFO:teuthology.orchestra.run.smithi165.stdout:3927 punch from 0x35ebd47 to 0x35fabfa, (0xeeb3 bytes) 2024-09-11T05:52:03.864 INFO:teuthology.orchestra.run.smithi165.stdout:3928 punch from 0x344fbe6 to 0x34549dc, (0x4df6 bytes) 2024-09-11T05:52:03.877 INFO:teuthology.orchestra.run.smithi165.stdout:3929 trunc from 0x380bbdb to 0xdcbe635 2024-09-11T05:52:03.901 INFO:teuthology.orchestra.run.smithi165.stdout:3930 trunc from 0xdcbe635 to 0x8e14e0 2024-09-11T05:52:03.924 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:03.919+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:03.978 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:03.979 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:03.979 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:03.979 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:03.979 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:04.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:04.049 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:04.047+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:04.119 INFO:teuthology.orchestra.run.smithi165.stdout:3932 read 0x13cffb thru 0x13d5c8 (0x5ce bytes) 2024-09-11T05:52:04.120 INFO:teuthology.orchestra.run.smithi165.stdout:3933 clone 26 order 20 su 32768 sc 10 2024-09-11T05:52:04.125 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:04.124+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.179851272999997, 2024-09-11T05:52:04.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:52:04.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.790134788000003, 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:52:04.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:52:04.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:04.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.969283105999999, 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:04.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:04.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.670104047999999, 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.189 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90878 3.1e e64/23 3:7d08a1c9:::rbd_data.11242d2cfbf4.000000000000007c:head v 64'209, mlcod=64'207)", 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.4942787170000003, 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053596290000000003, 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:04.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678835+0000", 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4199999999999995e-07 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678840+0000", 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1270000000000002e-06 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678847+0000", 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-06 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678851+0000", 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5889999999999996e-06 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678918+0000", 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6451000000000005e-05 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678960+0000", 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2265000000000003e-05 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684084+0000", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051238810000000003 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684194+0000", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109744 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90881 3.a e64/28 3:538a4bad:::rbd_data.11242d2cfbf4.00000000000000bc:head v 64'253, mlcod=64'250)", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:04.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.4927236690000001, 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068461709999999999, 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680390+0000", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6600000000000005e-07 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680396+0000", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3940000000000001e-06 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680412+0000", 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5208000000000001e-05 2024-09-11T05:52:04.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680421+0000", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.628e-06 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680497+0000", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5315000000000007e-05 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680517+0000", 2024-09-11T05:52:04.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0437000000000001e-05 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687105+0000", 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00658787 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687235+0000", 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013045299999999999 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90886 3.1e e64/23 3:7a4cea0f:::rbd_data.11242d2cfbf4.00000000000001b5:head v 64'210, mlcod=64'207)", 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.4908765880000008, 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004920804, 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682237+0000", 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9199999999999999e-07 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682243+0000", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2920000000000001e-06 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682258+0000", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4062e-05 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682264+0000", 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9059999999999996e-06 2024-09-11T05:52:04.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682322+0000", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7989000000000003e-05 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682343+0000", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0307999999999999e-05 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687116+0000", 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047728049999999998 2024-09-11T05:52:04.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687157+0000", 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1449999999999998e-05 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:91434 3.1e e64/23 3:7835f8eb:::rbd_data.11242d2cfbf4.000000000000011c:head v 64'213, mlcod=64'211)", 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.9199000069999999, 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047778619999999999, 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253216+0000", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.117e-06 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253326+0000", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110404 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253370+0000", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3927999999999997e-05 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253393+0000", 2024-09-11T05:52:04.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2758000000000001e-05 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253552+0000", 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159092 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253617+0000", 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5170000000000001e-05 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257880+0000", 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042626649999999997 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257991+0000", 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110728 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.0 epoch 68/56 rep_tid 208 entries 68'364 (0'0) error 3:00542219:::rbd_data.112419b96064.0000000000000063:head by client.4388.0:94505 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'362)", 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.281114085, 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058888120000000002, 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892001+0000", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.153e-06 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892024+0000", 2024-09-11T05:52:04.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3025999999999999e-05 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892091+0000", 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6669999999999997e-05 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892133+0000", 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2614000000000002e-05 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892369+0000", 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000236126 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897887+0000", 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055182230000000001 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.10 epoch 68/28 rep_tid 2085 entries 68'195 (0'0) error 3:0c91dc8e:::rbd_data.112419b96064.000000000000005e:head by client.4388.0:94500 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'193)", 2024-09-11T05:52:04.215 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2810325549999999, 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058577289999999999, 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:04.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892082+0000", 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.88e-06 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892104+0000", 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691999999999998e-05 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892157+0000", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3260000000000002e-05 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892199+0000", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1572000000000001e-05 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892924+0000", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072574599999999999 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897938+0000", 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050135789999999998 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.0 epoch 68/56 rep_tid 209 entries 68'365 (0'0) error 3:051e595e:::rbd_data.112419b96064.0000000000000067:head by client.4388.0:94509 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'362)", 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2799529220000001, 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048161080000000004, 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893162+0000", 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.866e-06 2024-09-11T05:52:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893184+0000", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2342000000000001e-05 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893236+0000", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1981000000000003e-05 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893282+0000", 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6372000000000003e-05 2024-09-11T05:52:04.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893441+0000", 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015828500000000001 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897976+0000", 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004535262 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106996 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'581, mlcod=70'576)", 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.55172026500000004, 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050416039999999999, 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621400+0000", 2024-09-11T05:52:04.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8839999999999997e-06 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621412+0000", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1491999999999999e-05 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621418+0000", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621477+0000", 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9117000000000003e-05 2024-09-11T05:52:04.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621499+0000", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1480000000000001e-05 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.626388+0000", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048888309999999997 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.626434+0000", 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6520999999999997e-05 2024-09-11T05:52:04.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106998 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'583, mlcod=70'578)", 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.55014460700000001, 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061126100000000001, 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622969+0000", 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2300000000000001e-06 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622981+0000", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1968e-05 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622999+0000", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8054999999999999e-05 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623010+0000", 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0351e-05 2024-09-11T05:52:04.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623112+0000", 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010228700000000001 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623143+0000", 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0630000000000003e-05 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629025+0000", 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058822600000000003 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629081+0000", 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5828999999999997e-05 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106999 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'584, mlcod=70'578)", 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:04.231 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.54937312500000002, 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005373694, 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623742+0000", 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1399999999999998e-06 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623784+0000", 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1739000000000001e-05 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623812+0000", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8697999999999999e-05 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623832+0000", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9505999999999999e-05 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623953+0000", 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121506 2024-09-11T05:52:04.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623978+0000", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4720999999999999e-05 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629084+0000", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051060189999999998 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629113+0000", 2024-09-11T05:52:04.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9365000000000001e-05 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109715 3.0 e70/56 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head v 70'420, mlcod=70'418)", 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.26652506500000001, 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050791059999999999, 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906590+0000", 2024-09-11T05:52:04.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906614+0000", 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3544999999999999e-05 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906663+0000", 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8915000000000002e-05 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906683+0000", 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907705+0000", 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010216369999999999 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907744+0000", 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9131999999999997e-05 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.911474+0000", 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037302669999999998 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.911667+0000", 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019260100000000001 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.240 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.26647932899999999, 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:04.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109736 3.1e e70/23 3:797e52f0:::rbd_data.11242c12cf9f.000000000000005d:head v 70'230, mlcod=70'227)", 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.914980+0000", 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.258132431, 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047287459999999998, 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.245 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914980+0000", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914980+0000", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914982+0000", 2024-09-11T05:52:04.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.311e-06 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914994+0000", 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2184000000000001e-05 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915016+0000", 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2623e-05 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915030+0000", 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3822000000000001e-05 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915105+0000", 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4456000000000007e-05 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915126+0000", 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1379999999999999e-05 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.919658+0000", 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004531607 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.919709+0000", 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1363000000000003e-05 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109735 3.10 e70/28 3:0d5a9039:::rbd_data.11242c12cf9f.000000000000005c:head v 70'627, mlcod=70'625)", 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.25787933899999999, 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048687849999999996, 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915234+0000", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.78e-07 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915241+0000", 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9890000000000003e-06 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915253+0000", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1892999999999999e-05 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915261+0000", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6639999999999998e-06 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915787+0000", 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052653900000000002 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915815+0000", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7917000000000001e-05 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.919995+0000", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041799139999999999 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.920102+0000", 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010699100000000001 2024-09-11T05:52:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109742 3.10 e70/28 3:08057856:::rbd_data.11242c12cf9f.0000000000000063:head v 70'628, mlcod=70'625)", 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.25505731799999998, 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048086930000000002, 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918057+0000", 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.528e-06 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918068+0000", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0978999999999999e-05 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918095+0000", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7461999999999999e-05 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918110+0000", 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4409e-05 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918294+0000", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000184005 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918330+0000", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5994999999999997e-05 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922827+0000", 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044972930000000003 2024-09-11T05:52:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922864+0000", 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7021999999999999e-05 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109743 3.0 e70/56 3:035834ed:::rbd_data.11242c12cf9f.000000000000006e:head v 70'421, mlcod=70'419)", 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.918068+0000", 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.25504515300000002, 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047554870000000001, 2024-09-11T05:52:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918068+0000", 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918068+0000", 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918069+0000", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.345e-06 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918086+0000", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7512000000000001e-05 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918104+0000", 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8029e-05 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918114+0000", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8870000000000004e-06 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918261+0000", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146453 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918306+0000", 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5225999999999997e-05 2024-09-11T05:52:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922771+0000", 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044653230000000002 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922823+0000", 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1712000000000002e-05 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.262 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:04.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:04.264 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:04.264+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:04.365 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:04.364+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:04.388 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T05:52:04.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:52:04.391 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:04.391 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:04.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:04.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T05:52:04.492 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone25 from 0x8e14e0 (overlap 0x8e14e0) to 0x1c6a93 2024-09-11T05:52:04.493 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:04.492+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:04.557 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:04.557 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:04.557 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:04.557 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:04.557 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:04.566 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:04.563+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:04.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:04.667 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:04.663+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.639605766000003, 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:52:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.398018383, 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:52:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.397407557000001, 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.39738315, 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:52:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.397372831999999, 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:52:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:52:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394769217, 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394758775, 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:52:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394747068000001, 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:04.786 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394735988000001, 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:52:04.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394700893, 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:52:04.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:52:04.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394660386999998, 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:52:04.795 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:52:04.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394626591000002, 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:52:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394587688000001, 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394501186999999, 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394483925999999, 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394177851999999, 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:52:04.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394118078999998, 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:52:04.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394096436000002, 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:52:04.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.394075232999999, 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:52:04.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:04.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:52:04.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.393030442000001, 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:04.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:04.837 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:04.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:04.839 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:04.836+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:04.939 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:04.936+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:04.971 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:04.972 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:04.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:05.072 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-09-11T05:52:05.072 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:05.072+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:05.084 INFO:teuthology.orchestra.run.smithi165.stdout:3935 punch from 0x2096c2 to 0x21431d, (0xac5b bytes) 2024-09-11T05:52:05.091 INFO:teuthology.orchestra.run.smithi165.stdout:3936 punch from 0x7861c8 to 0x795cb0, (0xfae8 bytes) 2024-09-11T05:52:05.091 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:05.091 INFO:teuthology.orchestra.run.smithi165.stdout:3938 read 0x8ac613 thru 0x8b7528 (0xaf16 bytes) 2024-09-11T05:52:05.093 INFO:teuthology.orchestra.run.smithi165.stdout:3939 read 0x122f50 thru 0x12a13c (0x71ed bytes) 2024-09-11T05:52:05.124 INFO:teuthology.orchestra.run.smithi165.stdout:3940 write 0xae93f82 thru 0xae96e24 (0x2ea3 bytes) 2024-09-11T05:52:05.134 INFO:teuthology.orchestra.run.smithi165.stdout:3941 writesame 0x54447d5 thru 0x54501d8 data_size 0xa(0xba04 bytes) 2024-09-11T05:52:05.141 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:05.139+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:05.148 INFO:teuthology.orchestra.run.smithi165.stdout:3942 read 0x74fcfb1 thru 0x7505251 (0x82a1 bytes) 2024-09-11T05:52:05.148 INFO:teuthology.orchestra.run.smithi165.stdout:3944 read 0x9be7f33 thru 0x9be91d1 (0x129f bytes) 2024-09-11T05:52:05.148 INFO:teuthology.orchestra.run.smithi165.stdout:3945 read 0x3644bfd thru 0x364efb4 (0xa3b8 bytes) 2024-09-11T05:52:05.148 INFO:teuthology.orchestra.run.smithi165.stdout:3948 writesame 0x60bc270 thru 0x60cbcac data_size 0x1d(0xfa3d bytes) 2024-09-11T05:52:05.159 INFO:teuthology.orchestra.run.smithi165.stdout:3949 read 0x170da58 thru 0x1717e02 (0xa3ab bytes) 2024-09-11T05:52:05.159 INFO:teuthology.orchestra.run.smithi165.stdout:3950 read 0x65a4f9c thru 0x65b2c0e (0xdc73 bytes) 2024-09-11T05:52:05.160 INFO:teuthology.orchestra.run.smithi165.stdout:3951 read 0xcdf68f thru 0xce4f8c (0x58fe bytes) 2024-09-11T05:52:05.160 INFO:teuthology.orchestra.run.smithi165.stdout:3952 read 0x5a6df2f thru 0x5a744f6 (0x65c8 bytes) 2024-09-11T05:52:05.168 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:05.168 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:05.168 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:05.168 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:05.169 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:05.181 INFO:teuthology.orchestra.run.smithi165.stdout:3954 writesame 0xe4fe445 thru 0xe501bec data_size 0x1a(0x37a8 bytes) 2024-09-11T05:52:05.181 INFO:teuthology.orchestra.run.smithi165.stdout:3955 read 0x49967d thru 0x4a0281 (0x6c05 bytes) 2024-09-11T05:52:05.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:05.183 INFO:teuthology.orchestra.run.smithi165.stdout:3956 write 0x41b6c51 thru 0x41c6b64 (0xff14 bytes) 2024-09-11T05:52:05.196 INFO:teuthology.orchestra.run.smithi165.stdout:3957 writesame 0x18dbc18 thru 0x18df033 data_size 0x3a(0x341c bytes) 2024-09-11T05:52:05.202 INFO:teuthology.orchestra.run.smithi165.stdout:3958 write 0xd125837 thru 0xd12a948 (0x5112 bytes) 2024-09-11T05:52:05.213 INFO:teuthology.orchestra.run.smithi165.stdout:3959 punch from 0xa1b824c to 0xa1bdba1, (0x5955 bytes) 2024-09-11T05:52:05.213 INFO:teuthology.orchestra.run.smithi165.stdout:3960 write 0x3a01749 thru 0x3a0e60e (0xcec6 bytes) 2024-09-11T05:52:05.223 INFO:teuthology.orchestra.run.smithi165.stdout:3961 punch from 0x3155ec1 to 0x3158611, (0x2750 bytes) 2024-09-11T05:52:05.223 INFO:teuthology.orchestra.run.smithi165.stdout:3962 writesame 0xc239efc thru 0xc23be3a data_size 0x1(0x1f3f bytes) 2024-09-11T05:52:05.233 INFO:teuthology.orchestra.run.smithi165.stdout:3963 writesame 0x886d821 thru 0x88745fe data_size 0xe(0x6dde bytes) 2024-09-11T05:52:05.242 INFO:teuthology.orchestra.run.smithi165.stdout:3964 punch from 0x5d5ce5f to 0x5d6ad1e, (0xdebf bytes) 2024-09-11T05:52:05.242 INFO:teuthology.orchestra.run.smithi165.stdout:3965 writesame 0xac3aeda thru 0xac491da data_size 0x21(0xe301 bytes) 2024-09-11T05:52:05.242 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:05.240+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:05.253 INFO:teuthology.orchestra.run.smithi165.stdout:3966 write 0x8df589 thru 0x8e07be (0x1236 bytes) 2024-09-11T05:52:05.312 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508169024000001, 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:05.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.317 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508142045, 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:05.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508111786000001, 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:05.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508066501, 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508033635, 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:05.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:05.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.508005580000001, 2024-09-11T05:52:05.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.339 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507978513999999, 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:05.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507928513, 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.344 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:05.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.348 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507896029999999, 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507867560999999, 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.353 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.357 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507838681999999, 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:05.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:05.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507781864, 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:05.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507750831999999, 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:05.366 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:05.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507722813000001, 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:05.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507695884, 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.375 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:05.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:05.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:05.379 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507669318, 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:05.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507632106999999, 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:05.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:05.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507603553999999, 2024-09-11T05:52:05.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:05.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:05.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507555093000001, 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:05.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.507516477999999, 2024-09-11T05:52:05.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:05.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:05.402 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:05.402 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:05.402 INFO:teuthology.orchestra.run.smithi165.stdout:3967 read 0xa97b53d thru 0xa98a958 (0xf41c bytes) 2024-09-11T05:52:05.403 INFO:teuthology.orchestra.run.smithi165.stdout:3969 write 0xede0548 thru 0xede1286 (0xd3f bytes) 2024-09-11T05:52:05.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:05.404 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:05.404 INFO:teuthology.orchestra.run.smithi165.stdout:3971 write 0xaea87d6 thru 0xaeb6a18 (0xe243 bytes) 2024-09-11T05:52:05.404 INFO:teuthology.orchestra.run.smithi165.stdout:3972 trunc from 0xede1287 to 0xd74df1f 2024-09-11T05:52:05.404 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:05.405 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:05.405 INFO:teuthology.orchestra.run.smithi165.stdout:3975 writesame 0x5d45a7 thru 0x5d8c1a data_size 0x36(0x4674 bytes) 2024-09-11T05:52:05.405 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:52:05.405 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T05:52:05.405 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:05.400+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:05.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T05:52:05.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:05.520 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:05.520+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:05.605 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:05.599+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:05.616 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:05.616 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:05.616 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:05.616 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:05.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:05.656 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:05.656 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:05.672 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:05.673 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:05.690 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:05.691 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:05.705 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:05.704+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:05.746 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:05.747 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:05.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:05.805 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:05.804+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:05.907 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:05.903+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:05.910 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:05.910 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:05.920 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:05.920 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:05.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:05.935 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:05.965 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:05.965 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:05.993 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:05.993 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:06.007 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:06.004+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:06.022 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:06.022 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:06.058 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.058 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:06.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:06.108 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:06.103+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:06.208 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:06.208+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:06.241 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.241 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:06.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:06.310 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:06.307+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:06.411 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:06.408+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:06.425 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.425 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:06.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:06.512 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T05:52:06.513 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:06.512+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:06.539 INFO:teuthology.orchestra.run.smithi165.stdout:3976 write 0x67b32d3 thru 0x67bb943 (0x8671 bytes) 2024-09-11T05:52:06.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T05:52:06.550 INFO:teuthology.orchestra.run.smithi165.stdout:3978 write 0x7bdb9f4 thru 0x7be0b1b (0x5128 bytes) 2024-09-11T05:52:06.558 INFO:teuthology.orchestra.run.smithi165.stdout:3979 write 0x3dd5f1e thru 0x3de0571 (0xa654 bytes) 2024-09-11T05:52:06.570 INFO:teuthology.orchestra.run.smithi165.stdout:3980 write 0xcbccd12 thru 0xcbd90bc (0xc3ab bytes) 2024-09-11T05:52:06.578 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.578 INFO:teuthology.orchestra.run.smithi165.stdout:3982 writesame 0xa2dc3fe thru 0xa2e0bb8 data_size 0x1(0x47bb bytes) 2024-09-11T05:52:06.595 INFO:teuthology.orchestra.run.smithi165.stdout:3983 punch from 0x9fd23f2 to 0x9fd8cbc, (0x68ca bytes) 2024-09-11T05:52:06.595 INFO:teuthology.orchestra.run.smithi165.stdout:3984 writesame 0x30e8267 thru 0x30ece91 data_size 0x1f(0x4c2b bytes) 2024-09-11T05:52:06.603 INFO:teuthology.orchestra.run.smithi165.stdout:3985 read 0x901a748 thru 0x90269dd (0xc296 bytes) 2024-09-11T05:52:06.603 INFO:teuthology.orchestra.run.smithi165.stdout:3986 write 0x427ba3a thru 0x427bcea (0x2b1 bytes) 2024-09-11T05:52:06.612 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:06.612+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:06.624 INFO:teuthology.orchestra.run.smithi165.stdout:3987 trunc from 0xd74df1f to 0x4b85abd 2024-09-11T05:52:06.640 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.640 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:06.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:06.660 INFO:teuthology.orchestra.run.smithi165.stdout:3989 write 0x122d642 thru 0x1231a78 (0x4437 bytes) 2024-09-11T05:52:06.669 INFO:teuthology.orchestra.run.smithi165.stdout:3991 punch from 0x31dadb0 to 0x31e8249, (0xd499 bytes) 2024-09-11T05:52:06.673 INFO:teuthology.orchestra.run.smithi165.stdout:3993 trunc from 0x4b85abd to 0x763c4ef 2024-09-11T05:52:06.684 INFO:teuthology.orchestra.run.smithi165.stdout:3994 read 0x3649749 thru 0x36529bf (0x9277 bytes) 2024-09-11T05:52:06.684 INFO:teuthology.orchestra.run.smithi165.stdout:3996 punch from 0x5ee9fcb to 0x5ef9dc8, (0xfdfd bytes) 2024-09-11T05:52:06.685 INFO:teuthology.orchestra.run.smithi165.stdout:3997 punch from 0x4185581 to 0x418bfb8, (0x6a37 bytes) 2024-09-11T05:52:06.685 INFO:teuthology.orchestra.run.smithi165.stdout:3999 writesame 0x5d4a509 thru 0x5d4b9d2 data_size 0x3(0x14ca bytes) 2024-09-11T05:52:06.692 INFO:teuthology.orchestra.run.smithi165.stdout:4001 writesame 0x48e0568 thru 0x48eb721 data_size 0x3(0xb1ba bytes) 2024-09-11T05:52:06.706 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.706 INFO:teuthology.orchestra.run.smithi165.stdout:4005 read 0x1cea47e thru 0x1ced40f (0x2f92 bytes) 2024-09-11T05:52:06.707 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.713 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:06.711+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:06.721 INFO:teuthology.orchestra.run.smithi165.stdout:4007 write 0x9e33331 thru 0x9e35084 (0x1d54 bytes) 2024-09-11T05:52:06.728 INFO:teuthology.orchestra.run.smithi165.stdout:4009 read 0x70370ff thru 0x703e96f (0x7871 bytes) 2024-09-11T05:52:06.732 INFO:teuthology.orchestra.run.smithi165.stdout:4010 trunc from 0x9e35085 to 0x7ae4dc9 2024-09-11T05:52:06.754 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.754 INFO:teuthology.orchestra.run.smithi165.stdout:4012 read 0x3c129a6 thru 0x3c1683d (0x3e98 bytes) 2024-09-11T05:52:06.754 INFO:teuthology.orchestra.run.smithi165.stdout:4013 read 0x2d510f5 thru 0x2d52c3e (0x1b4a bytes) 2024-09-11T05:52:06.754 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.754 INFO:teuthology.orchestra.run.smithi165.stdout:4015 writesame 0x1493d38 thru 0x1497e82 data_size 0x1(0x414b bytes) 2024-09-11T05:52:06.769 INFO:teuthology.orchestra.run.smithi165.stdout:4016 punch from 0x421bfc1 to 0x42292b1, (0xd2f0 bytes) 2024-09-11T05:52:06.769 INFO:teuthology.orchestra.run.smithi165.stdout:4018 read 0x3cde943 thru 0x3ce7048 (0x8706 bytes) 2024-09-11T05:52:06.781 INFO:teuthology.orchestra.run.smithi165.stdout:4020 writesame 0x81cf4e6 thru 0x81d7594 data_size 0x4f(0x80af bytes) 2024-09-11T05:52:06.797 INFO:teuthology.orchestra.run.smithi165.stdout:4022 writesame 0xadc4b02 thru 0xadd1a8a data_size 0x1(0xcf89 bytes) 2024-09-11T05:52:06.797 INFO:teuthology.orchestra.run.smithi165.stdout:4023 write 0x3d0c779 thru 0x3d13047 (0x68cf bytes) 2024-09-11T05:52:06.800 INFO:teuthology.orchestra.run.smithi165.stdout:4026 read 0x6fed637 thru 0x6ff2057 (0x4a21 bytes) 2024-09-11T05:52:06.801 INFO:teuthology.orchestra.run.smithi165.stdout:4027 read 0x3e5895e thru 0x3e649d3 (0xc076 bytes) 2024-09-11T05:52:06.811 INFO:teuthology.orchestra.run.smithi165.stdout:4031 write 0xb8e1925 thru 0xb8f1550 (0xfc2c bytes) 2024-09-11T05:52:06.814 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:06.812+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:06.816 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.816 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:06.822 INFO:teuthology.orchestra.run.smithi165.stdout:4034 punch from 0x85991e0 to 0x85a44e7, (0xb307 bytes) 2024-09-11T05:52:06.822 INFO:teuthology.orchestra.run.smithi165.stdout:4036 read 0x84d171e thru 0x84dc8bb (0xb19e bytes) 2024-09-11T05:52:06.822 INFO:teuthology.orchestra.run.smithi165.stdout:4039 write 0x1dde429 thru 0x1de1adf (0x36b7 bytes) 2024-09-11T05:52:06.829 INFO:teuthology.orchestra.run.smithi165.stdout:4040 write 0x5fcf3ba thru 0x5fd2aac (0x36f3 bytes) 2024-09-11T05:52:06.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:06.835 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.836 INFO:teuthology.orchestra.run.smithi165.stdout:4042 write 0xb44be38 thru 0xb4574b0 (0xb679 bytes) 2024-09-11T05:52:06.844 INFO:teuthology.orchestra.run.smithi165.stdout:4043 write 0xc92b0e thru 0xc9ad25 (0x8218 bytes) 2024-09-11T05:52:06.866 INFO:teuthology.orchestra.run.smithi165.stdout:4044 write 0xde272de thru 0xde35be8 (0xe90b bytes) 2024-09-11T05:52:06.877 INFO:teuthology.orchestra.run.smithi165.stdout:4046 write 0xc198842 thru 0xc1a3b9b (0xb35a bytes) 2024-09-11T05:52:06.887 INFO:teuthology.orchestra.run.smithi165.stdout:4047 write 0x41e9910 thru 0x41f8cf7 (0xf3e8 bytes) 2024-09-11T05:52:06.895 INFO:teuthology.orchestra.run.smithi165.stdout:4048 write 0x471dfcb thru 0x471ec10 (0xc46 bytes) 2024-09-11T05:52:06.903 INFO:teuthology.orchestra.run.smithi165.stdout:4049 read 0x80bc17d thru 0x80c0fbf (0x4e43 bytes) 2024-09-11T05:52:06.903 INFO:teuthology.orchestra.run.smithi165.stdout:4050 read 0xb3879d4 thru 0xb392b9b (0xb1c8 bytes) 2024-09-11T05:52:06.903 INFO:teuthology.orchestra.run.smithi165.stdout:4051 write 0x1e6314f thru 0x1e7091d (0xd7cf bytes) 2024-09-11T05:52:06.914 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:06.914 INFO:teuthology.orchestra.run.smithi165.stdout:4053 read 0xc42251e thru 0xc42ee14 (0xc8f7 bytes) 2024-09-11T05:52:06.914 INFO:teuthology.orchestra.run.smithi165.stdout:4055 read 0x20c674e thru 0x20d2a28 (0xc2db bytes) 2024-09-11T05:52:06.914 INFO:teuthology.orchestra.run.smithi165.stdout:4057 read 0xb872b8f thru 0xb874388 (0x17fa bytes) 2024-09-11T05:52:06.915 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:06.911+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:06.924 INFO:teuthology.orchestra.run.smithi165.stdout:4058 write 0xe5a2a89 thru 0xe5ab2d3 (0x884b bytes) 2024-09-11T05:52:06.929 INFO:teuthology.orchestra.run.smithi165.stdout:4059 read 0x8266a35 thru 0x827556f (0xeb3b bytes) 2024-09-11T05:52:06.929 INFO:teuthology.orchestra.run.smithi165.stdout:4060 write 0xcabf46a thru 0xcac996a (0xa501 bytes) 2024-09-11T05:52:06.939 INFO:teuthology.orchestra.run.smithi165.stdout:4061 write 0xd5c33 thru 0xdca81 (0x6e4f bytes) 2024-09-11T05:52:06.942 INFO:teuthology.orchestra.run.smithi165.stdout:4063 write 0x48d8634 thru 0x48e16c0 (0x908d bytes) 2024-09-11T05:52:06.951 INFO:teuthology.orchestra.run.smithi165.stdout:4064 write 0xb75059a thru 0xb75da04 (0xd46b bytes) 2024-09-11T05:52:06.955 INFO:teuthology.orchestra.run.smithi165.stdout:4066 read 0x1781ad7 thru 0x1786a3f (0x4f69 bytes) 2024-09-11T05:52:06.969 INFO:teuthology.orchestra.run.smithi165.stdout:4067 trunc from 0xe5ab2d4 to 0x439cc40 2024-09-11T05:52:06.986 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:06.986 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:07.004 INFO:teuthology.orchestra.run.smithi165.stdout:4068 punch from 0x8bd427 to 0x8cb84b, (0xe424 bytes) 2024-09-11T05:52:07.005 INFO:teuthology.orchestra.run.smithi165.stdout:4069 read 0x650cb9 thru 0x6604d2 (0xf81a bytes) 2024-09-11T05:52:07.007 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:07.016 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:07.016+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:07.016 INFO:teuthology.orchestra.run.smithi165.stdout:4072 writesame 0x5177f3b thru 0x5187ef6 data_size 0x1a(0xffbc bytes) 2024-09-11T05:52:07.016 INFO:teuthology.orchestra.run.smithi165.stdout:4073 punch from 0x1d1df3f to 0x1d2234e, (0x440f bytes) 2024-09-11T05:52:07.016 INFO:teuthology.orchestra.run.smithi165.stdout:4074 punch from 0x29c3697 to 0x29c404b, (0x9b4 bytes) 2024-09-11T05:52:07.022 INFO:teuthology.orchestra.run.smithi165.stdout:4076 trunc from 0x5187ef7 to 0x993a229 2024-09-11T05:52:07.030 INFO:teuthology.orchestra.run.smithi165.stdout:4077 write 0x1235a41 thru 0x12455e5 (0xfba5 bytes) 2024-09-11T05:52:07.039 INFO:teuthology.orchestra.run.smithi165.stdout:4079 trunc from 0x993a229 to 0x78b2c6d 2024-09-11T05:52:07.061 INFO:teuthology.orchestra.run.smithi165.stdout:4080 write 0xc20dcfd thru 0xc21c28b (0xe58f bytes) 2024-09-11T05:52:07.076 INFO:teuthology.orchestra.run.smithi165.stdout:4083 write 0xcee023e thru 0xcee9203 (0x8fc6 bytes) 2024-09-11T05:52:07.094 INFO:teuthology.orchestra.run.smithi165.stdout:4084 write 0xea09cc7 thru 0xea17752 (0xda8c bytes) 2024-09-11T05:52:07.102 INFO:teuthology.orchestra.run.smithi165.stdout:4085 trunc from 0xea17753 to 0xca9e58a 2024-09-11T05:52:07.116 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:07.116+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:07.136 INFO:teuthology.orchestra.run.smithi165.stdout:4087 write 0xe6905f6 thru 0xe69959e (0x8fa9 bytes) 2024-09-11T05:52:07.142 INFO:teuthology.orchestra.run.smithi165.stdout:4088 read 0xe14b0ad thru 0xe155693 (0xa5e7 bytes) 2024-09-11T05:52:07.142 INFO:teuthology.orchestra.run.smithi165.stdout:4089 read 0xbf7f325 thru 0xbf8eb34 (0xf810 bytes) 2024-09-11T05:52:07.143 INFO:teuthology.orchestra.run.smithi165.stdout:4091 read 0x77fa808 thru 0x7804883 (0xa07c bytes) 2024-09-11T05:52:07.147 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.147 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:07.161 INFO:teuthology.orchestra.run.smithi165.stdout:4092 trunc from 0xe69959f to 0x494bc28 2024-09-11T05:52:07.208 INFO:teuthology.orchestra.run.smithi165.stdout:4093 write 0xdd97700 thru 0xdd9bef6 (0x47f7 bytes) 2024-09-11T05:52:07.214 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:07.214 INFO:teuthology.orchestra.run.smithi165.stdout:4095 read 0x220b5b3 thru 0x220d12f (0x1b7d bytes) 2024-09-11T05:52:07.214 INFO:teuthology.orchestra.run.smithi165.stdout:4096 read 0x316aa6e thru 0x316e3eb (0x397e bytes) 2024-09-11T05:52:07.214 INFO:teuthology.orchestra.run.smithi165.stdout:4097 write 0x33df86b thru 0x33e8003 (0x8799 bytes) 2024-09-11T05:52:07.217 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:07.216+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:07.220 INFO:teuthology.orchestra.run.smithi165.stdout:4098 read 0x935dd5d thru 0x936140c (0x36b0 bytes) 2024-09-11T05:52:07.221 INFO:teuthology.orchestra.run.smithi165.stdout:4099 trunc from 0xdd9bef7 to 0xdfaacaf 2024-09-11T05:52:07.229 INFO:teuthology.orchestra.run.smithi165.stdout:4100 writesame 0x3c887e9 thru 0x3c8bd3a data_size 0x5b(0x3552 bytes) 2024-09-11T05:52:07.250 INFO:teuthology.orchestra.run.smithi165.stdout:4101 trunc from 0xdfaacaf to 0x3f0ef5e 2024-09-11T05:52:07.291 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.291 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.299 INFO:teuthology.orchestra.run.smithi165.stdout:4102 writesame 0x776f5a9 thru 0x777af99 data_size 0x2b(0xb9f1 bytes) 2024-09-11T05:52:07.299 INFO:teuthology.orchestra.run.smithi165.stdout:4103 write 0x45c446a thru 0x45d0916 (0xc4ad bytes) 2024-09-11T05:52:07.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:07.319 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:07.316+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:07.329 INFO:teuthology.orchestra.run.smithi165.stdout:4104 write 0xe32a1f3 thru 0xe32dca3 (0x3ab1 bytes) 2024-09-11T05:52:07.334 INFO:teuthology.orchestra.run.smithi165.stdout:4106 punch from 0x7d797e2 to 0x7d84bcb, (0xb3e9 bytes) 2024-09-11T05:52:07.366 INFO:teuthology.orchestra.run.smithi165.stdout:4108 trunc from 0xe32dca4 to 0x11c7685 2024-09-11T05:52:07.394 INFO:teuthology.orchestra.run.smithi165.stdout:4109 read 0xf8e60 thru 0xfdb8b (0x4d2c bytes) 2024-09-11T05:52:07.396 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:07.409 INFO:teuthology.orchestra.run.smithi165.stdout:4111 writesame 0x1c64d39 thru 0x1c65b9e data_size 0x13(0xe66 bytes) 2024-09-11T05:52:07.419 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:07.415+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:07.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:52:07.431 INFO:teuthology.orchestra.run.smithi165.stdout:4112 write 0x64ccc33 thru 0x64d6f75 (0xa343 bytes) 2024-09-11T05:52:07.437 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.437 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.438 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:07.438 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:07.438 INFO:teuthology.orchestra.run.smithi165.stdout:4116 punch from 0x21a8272 to 0x21adfbc, (0x5d4a bytes) 2024-09-11T05:52:07.438 INFO:teuthology.orchestra.run.smithi165.stdout:4117 punch from 0x3e0bc74 to 0x3e0d118, (0x14a4 bytes) 2024-09-11T05:52:07.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:07.453 INFO:teuthology.orchestra.run.smithi165.stdout:4119 trunc from 0x64d6f76 to 0x11e873 2024-09-11T05:52:07.515 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T05:52:07.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T05:52:07.625 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.625 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.627 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:07.520+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:07.631 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:07.628+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:07.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:07.721 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:07.715+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:07.758 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.758 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:07.822 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:07.820+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:07.923 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:07.920+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:07.931 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:07.931 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:07.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:08.025 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:08.024+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:08.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:08.070 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:08.070 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:08.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:08.125 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:08.124+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:08.199 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:08.199 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:08.205 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:08.205 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:08.217 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:08.217 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:08.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:08.226 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:08.224+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:08.233 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:08.233 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:08.251 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:08.251 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:08.268 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:08.269 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:08.286 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:08.286 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:08.304 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:08.304 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:08.323 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:08.324 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:08.327 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:08.323+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:08.346 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:7100 3.a e34/28 3:524ed652:::rbd_data.11249b359231.0000000000000010:head v 34'30, mlcod=33'28)", 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.351946943000002, 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057515680000000003, 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993261+0000", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993264+0000", 2024-09-11T05:52:08.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3460000000000001e-06 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993280+0000", 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6263e-05 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993323+0000", 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2812999999999998e-05 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993343+0000", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587000000000001e-05 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993586+0000", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000243141 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.993641+0000", 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5203e-05 2024-09-11T05:52:08.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.998867+0000", 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052260830000000003 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:17.999013+0000", 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014613199999999999 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:19396 3.1e e46/23 3:796bcbae:::rbd_data.11244d01bf21.0000000000000061:head v 46'57, mlcod=45'55)", 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.962230458000001, 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059860010000000003, 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:08.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382978+0000", 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.382981+0000", 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.943e-06 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383267+0000", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028643599999999999 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383308+0000", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0796e-05 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383331+0000", 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2750000000000001e-05 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383574+0000", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024276500000000001 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.383630+0000", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6214000000000001e-05 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388828+0000", 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051976349999999999 2024-09-11T05:52:08.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:31.388964+0000", 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013646200000000001 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.141378776, 2024-09-11T05:52:08.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.842199718, 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:08.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:08.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:08.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90878 3.1e e64/23 3:7d08a1c9:::rbd_data.11242d2cfbf4.000000000000007c:head v 64'209, mlcod=64'207)", 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.666374386999999, 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053596290000000003, 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678834+0000", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678835+0000", 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4199999999999995e-07 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678840+0000", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1270000000000002e-06 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678847+0000", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9299999999999997e-06 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678851+0000", 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5889999999999996e-06 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678918+0000", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6451000000000005e-05 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.678960+0000", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2265000000000003e-05 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684084+0000", 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051238810000000003 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.684194+0000", 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109744 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90881 3.a e64/28 3:538a4bad:::rbd_data.11242d2cfbf4.00000000000000bc:head v 64'253, mlcod=64'250)", 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.664819338999999, 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068461709999999999, 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.369 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680389+0000", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680390+0000", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6600000000000005e-07 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680396+0000", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3940000000000001e-06 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680412+0000", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5208000000000001e-05 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680421+0000", 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.628e-06 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680497+0000", 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5315000000000007e-05 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.680517+0000", 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0437000000000001e-05 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687105+0000", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00658787 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687235+0000", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013045299999999999 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:90886 3.1e e64/23 3:7a4cea0f:::rbd_data.11242d2cfbf4.00000000000001b5:head v 64'210, mlcod=64'207)", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:08.373 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.662972258, 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004920804, 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682236+0000", 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682237+0000", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9199999999999999e-07 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682243+0000", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2920000000000001e-06 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682258+0000", 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4062e-05 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682264+0000", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9059999999999996e-06 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682322+0000", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7989000000000003e-05 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.682343+0000", 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0307999999999999e-05 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687116+0000", 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047728049999999998 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:55.687157+0000", 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1449999999999998e-05 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.377 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:91434 3.1e e64/23 3:7835f8eb:::rbd_data.11242d2cfbf4.000000000000011c:head v 64'213, mlcod=64'211)", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.091995677, 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047778619999999999, 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253213+0000", 2024-09-11T05:52:08.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253216+0000", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.117e-06 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253326+0000", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110404 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253370+0000", 2024-09-11T05:52:08.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3927999999999997e-05 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253393+0000", 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2758000000000001e-05 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253552+0000", 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159092 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.253617+0000", 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5170000000000001e-05 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257880+0000", 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042626649999999997 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:56.257991+0000", 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110728 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.0 epoch 68/56 rep_tid 208 entries 68'364 (0'0) error 3:00542219:::rbd_data.112419b96064.0000000000000063:head by client.4388.0:94505 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'362)", 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.4532097549999996, 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058888120000000002, 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.891999+0000", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892001+0000", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.153e-06 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892024+0000", 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3025999999999999e-05 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892091+0000", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6669999999999997e-05 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892133+0000", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2614000000000002e-05 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892369+0000", 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000236126 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897887+0000", 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055182230000000001 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.10 epoch 68/28 rep_tid 2085 entries 68'195 (0'0) error 3:0c91dc8e:::rbd_data.112419b96064.000000000000005e:head by client.4388.0:94500 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'193)", 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.4531282250000004, 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058577289999999999, 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892080+0000", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892082+0000", 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.88e-06 2024-09-11T05:52:08.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892104+0000", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691999999999998e-05 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892157+0000", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3260000000000002e-05 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892199+0000", 2024-09-11T05:52:08.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1572000000000001e-05 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.892924+0000", 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072574599999999999 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897938+0000", 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050135789999999998 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "pg_update_log_missing(3.0 epoch 68/56 rep_tid 209 entries 68'365 (0'0) error 3:051e595e:::rbd_data.112419b96064.0000000000000067:head by client.4388.0:94509 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 68'362)", 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.4520485919999997, 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048161080000000004, 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893160+0000", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893162+0000", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.866e-06 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893184+0000", 2024-09-11T05:52:08.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2342000000000001e-05 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893236+0000", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1981000000000003e-05 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893282+0000", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6372000000000003e-05 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.893441+0000", 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015828500000000001 2024-09-11T05:52:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:00.897976+0000", 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004535262 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106996 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'581, mlcod=70'576)", 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.7238159350000002, 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050416039999999999, 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621393+0000", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621400+0000", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8839999999999997e-06 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621412+0000", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1491999999999999e-05 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621418+0000", 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5719999999999999e-06 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621477+0000", 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9117000000000003e-05 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.621499+0000", 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1480000000000001e-05 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.626388+0000", 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048888309999999997 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.626434+0000", 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6520999999999997e-05 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106998 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'583, mlcod=70'578)", 2024-09-11T05:52:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.722240277, 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061126100000000001, 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622968+0000", 2024-09-11T05:52:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622969+0000", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2300000000000001e-06 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622981+0000", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1968e-05 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.622999+0000", 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8054999999999999e-05 2024-09-11T05:52:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623010+0000", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0351e-05 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623112+0000", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010228700000000001 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623143+0000", 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0630000000000003e-05 2024-09-11T05:52:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629025+0000", 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058822600000000003 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629081+0000", 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5828999999999997e-05 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:106999 3.10 e70/28 3:0e79f892:::rbd_object_map.11242c12cf9f:head v 70'584, mlcod=70'578)", 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.7214687949999998, 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005373694, 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623740+0000", 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623742+0000", 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1399999999999998e-06 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623784+0000", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1739000000000001e-05 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623812+0000", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8697999999999999e-05 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623832+0000", 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9505999999999999e-05 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623953+0000", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121506 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.623978+0000", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4720999999999999e-05 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629084+0000", 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051060189999999998 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.629113+0000", 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9365000000000001e-05 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109715 3.0 e70/56 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head v 70'420, mlcod=70'418)", 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.4386207349999998, 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050791059999999999, 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906588+0000", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906590+0000", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906614+0000", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3544999999999999e-05 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906663+0000", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8915000000000002e-05 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906683+0000", 2024-09-11T05:52:08.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907705+0000", 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010216369999999999 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907744+0000", 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9131999999999997e-05 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.911474+0000", 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037302669999999998 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.911667+0000", 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019260100000000001 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.4385749990000001, 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:08.409 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:08.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109735 3.10 e70/28 3:0d5a9039:::rbd_data.11242c12cf9f.000000000000005c:head v 70'627, mlcod=70'625)", 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.4299750089999996, 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048687849999999996, 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915233+0000", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915234+0000", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.78e-07 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915241+0000", 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9890000000000003e-06 2024-09-11T05:52:08.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915253+0000", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1892999999999999e-05 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915261+0000", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6639999999999998e-06 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915787+0000", 2024-09-11T05:52:08.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052653900000000002 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.915815+0000", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7917000000000001e-05 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.919995+0000", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041799139999999999 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.920102+0000", 2024-09-11T05:52:08.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010699100000000001 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109742 3.10 e70/28 3:08057856:::rbd_data.11242c12cf9f.0000000000000063:head v 70'628, mlcod=70'625)", 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.4271529879999996, 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048086930000000002, 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918055+0000", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918057+0000", 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.528e-06 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918068+0000", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0978999999999999e-05 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918095+0000", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7461999999999999e-05 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918110+0000", 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4409e-05 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918294+0000", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000184005 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.918330+0000", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5994999999999997e-05 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922827+0000", 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044972930000000003 2024-09-11T05:52:08.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.922864+0000", 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7021999999999999e-05 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:117379 3.1e e71/23 3:7f1f63e0:::rbd_object_map.11242c12cf9f.000000000000001c:head v 71'245, mlcod=71'243)", 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.856389+0000", 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.4888192419999999, 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059765740000000001, 2024-09-11T05:52:08.422 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856389+0000", 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856389+0000", 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856402+0000", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856415+0000", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2829e-05 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856472+0000", 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7472000000000001e-05 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856502+0000", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9184999999999999e-05 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856580+0000", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8007e-05 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.856600+0000", 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0846e-05 2024-09-11T05:52:08.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.862285+0000", 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056845130000000004 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.862366+0000", 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0821000000000004e-05 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:117393 3.10 e71/28 3:0a1c090d:::rbd_data.11242c12cf9f.0000000000000026:head v 71'647, mlcod=71'644)", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.889078+0000", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.4561303090000002, 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051526009999999997, 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889078+0000", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889078+0000", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889080+0000", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4479999999999999e-06 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889090+0000", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0441999999999999e-05 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889109+0000", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8949000000000001e-05 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889119+0000", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0368e-05 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889212+0000", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2859e-05 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.889235+0000", 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3113e-05 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.894187+0000", 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049519380000000003 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.894231+0000", 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3483999999999999e-05 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.431 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:08.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:08.432 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:08.427+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:08.518 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T05:52:08.529 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:52:08.532 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:08.527+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:08.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T05:52:08.598 INFO:teuthology.orchestra.run.smithi165.stdout:4120 write 0x7c35ec6 thru 0x7c3a84d (0x4988 bytes) 2024-09-11T05:52:08.607 INFO:teuthology.orchestra.run.smithi165.stdout:4121 read 0x797b558 thru 0x7983065 (0x7b0e bytes) 2024-09-11T05:52:08.608 INFO:teuthology.orchestra.run.smithi165.stdout:4122 write 0x6cab54f thru 0x6cad301 (0x1db3 bytes) 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:121370 3.11 3:8862691a:::rbd_data.1124dfc336db.000000000000006b:head [set-alloc-hint object_size 1048576 write_size 1048576,write 898383~7603 in=7603b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:08.612470+0000", 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0014563759999999999, 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001537699, 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 121370 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612470+0000", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612470+0000", 2024-09-11T05:52:08.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612473+0000", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.14e-06 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612496+0000", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2792000000000001e-05 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612526+0000", 2024-09-11T05:52:08.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9906000000000001e-05 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612540+0000", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3927000000000001e-05 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612638+0000", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8148999999999998e-05 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.612842+0000", 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000203413 2024-09-11T05:52:08.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:08.613032+0000", 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019022700000000001 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:08.617 INFO:teuthology.orchestra.run.smithi165.stdout:4125 clone 27 order 22 su 16384 sc 6 2024-09-11T05:52:08.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:08.633 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:08.632+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:08.734 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:08.731+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:08.793 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:08.794 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:08.794 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:08.794 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:08.794 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:08.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:08.850 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:08.847+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:08.936 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:08.936+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.869984488999997, 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:52:08.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:08.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.628397106000001, 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:08.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:52:08.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:52:08.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:52:08.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.627786279999999, 2024-09-11T05:52:08.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.989 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:52:08.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:52:08.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.627761873000001, 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:52:08.993 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:08.997 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.627751555, 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:08.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:52:09.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625147940000002, 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625137498000001, 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:52:09.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625125790999999, 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.010 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:52:09.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625114710999998, 2024-09-11T05:52:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.018 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625079616000001, 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:09.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625039109999999, 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:52:09.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:52:09.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:52:09.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.625005313999999, 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:52:09.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624966410999999, 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.031 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.035 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624879910000001, 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624862649000001, 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.043 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624556575, 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.044 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:52:09.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:52:09.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.047 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624496801999999, 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:09.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:52:09.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:52:09.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624475158999999, 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:52:09.051 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:52:09.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.624453956, 2024-09-11T05:52:09.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:52:09.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.623409164999998, 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:52:09.059 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:09.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:09.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:09.063 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:09.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:09.065 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:09.064+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:09.166 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:09.167+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:09.193 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:09.193 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:09.193 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:09.193 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:09.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:09.267 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:09.264+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:09.330 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:09.330 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:09.331 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:09.331 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:09.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:09.388 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:09.388+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:09.469 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:09.468+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:09.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710876847, 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:09.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710849868, 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:09.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:09.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710819609, 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:09.525 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710774323999999, 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:09.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710741457999999, 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.538 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710713403, 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:09.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710686337, 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.543 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:09.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.547 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710636336, 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:09.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710603853, 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.552 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.556 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710575384, 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:09.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710546505, 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:09.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710489687000001, 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:09.565 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.569 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710458655, 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:09.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.573 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710430636, 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.574 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:09.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710403706999999, 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:09.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:09.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710377141, 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.583 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.71033993, 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:09.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.591 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710311377, 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:09.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710262916, 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:09.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.599 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.710224301, 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:09.604 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:09.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:09.606 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T05:52:09.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-11T05:52:09.607 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:09.599+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:09.671 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone26 from 0x7c3a84e (overlap 0x11e873) to 0x0 2024-09-11T05:52:09.712 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:09.712+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:09.787 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:09.787 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:09.787 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:09.787 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:09.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:09.806 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-09-11T05:52:09.807 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:09.803+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:09.815 INFO:teuthology.orchestra.run.smithi165.stdout:4126 writesame 0x1a567d7 thru 0x1a66072 data_size 0xe(0xf89c bytes) 2024-09-11T05:52:09.843 INFO:teuthology.orchestra.run.smithi165.stdout:4127 read 0x28272a0 thru 0x282f0b2 (0x7e13 bytes) 2024-09-11T05:52:09.859 INFO:teuthology.orchestra.run.smithi165.stdout:4129 write 0xbe63f2f thru 0xbe6b3e2 (0x74b4 bytes) 2024-09-11T05:52:09.872 INFO:teuthology.orchestra.run.smithi165.stdout:4130 read 0x93ec161 thru 0x93ee69d (0x253d bytes) 2024-09-11T05:52:09.872 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:09.872 INFO:teuthology.orchestra.run.smithi165.stdout:4132 read 0xaecdf95 thru 0xaed74fa (0x9566 bytes) 2024-09-11T05:52:09.873 INFO:teuthology.orchestra.run.smithi165.stdout:4133 read 0x6df2c83 thru 0x6df85df (0x595d bytes) 2024-09-11T05:52:09.873 INFO:teuthology.orchestra.run.smithi165.stdout:4134 write 0x707a60d thru 0x7083329 (0x8d1d bytes) 2024-09-11T05:52:09.899 INFO:teuthology.orchestra.run.smithi165.stdout:4137 write 0x4595c2a thru 0x45a3dca (0xe1a1 bytes) 2024-09-11T05:52:09.907 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:09.904+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:09.916 INFO:teuthology.orchestra.run.smithi165.stdout:4140 read 0x73960c0 thru 0x73a3606 (0xd547 bytes) 2024-09-11T05:52:09.916 INFO:teuthology.orchestra.run.smithi165.stdout:4142 read 0xcfbe3d thru 0xd03ea8 (0x806c bytes) 2024-09-11T05:52:09.925 INFO:teuthology.orchestra.run.smithi165.stdout:4143 write 0xc42ee9f thru 0xc43e955 (0xfab7 bytes) 2024-09-11T05:52:09.932 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:09.932 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:09.932 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:09.933 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:09.933 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:09.944 INFO:teuthology.orchestra.run.smithi165.stdout:4144 read 0x5ae387b thru 0x5aeda00 (0xa186 bytes) 2024-09-11T05:52:09.944 INFO:teuthology.orchestra.run.smithi165.stdout:4145 read 0x95f72ec thru 0x9601131 (0x9e46 bytes) 2024-09-11T05:52:09.944 INFO:teuthology.orchestra.run.smithi165.stdout:4147 read 0x1d796fb thru 0x1d7e58e (0x4e94 bytes) 2024-09-11T05:52:09.944 INFO:teuthology.orchestra.run.smithi165.stdout:4148 writesame 0x7ae1930 thru 0x7ae5b81 data_size 0xd(0x4252 bytes) 2024-09-11T05:52:09.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:09.954 INFO:teuthology.orchestra.run.smithi165.stdout:4149 write 0x3f80ca5 thru 0x3f88d2a (0x8086 bytes) 2024-09-11T05:52:09.959 INFO:teuthology.orchestra.run.smithi165.stdout:4151 write 0x86b9bb1 thru 0x86c4501 (0xa951 bytes) 2024-09-11T05:52:09.974 INFO:teuthology.orchestra.run.smithi165.stdout:4153 writesame 0x38942ff thru 0x38a3c4a data_size 0xa(0xf94c bytes) 2024-09-11T05:52:09.984 INFO:teuthology.orchestra.run.smithi165.stdout:4154 punch from 0x32607be to 0x326b305, (0xab47 bytes) 2024-09-11T05:52:09.984 INFO:teuthology.orchestra.run.smithi165.stdout:4155 write 0x110304b thru 0x110449c (0x1452 bytes) 2024-09-11T05:52:10.008 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:10.003+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:10.012 INFO:teuthology.orchestra.run.smithi165.stdout:4157 read 0x7314419 thru 0x73175eb (0x31d3 bytes) 2024-09-11T05:52:10.012 INFO:teuthology.orchestra.run.smithi165.stdout:4158 write 0x442dc05 thru 0x44390e4 (0xb4e0 bytes) 2024-09-11T05:52:10.026 INFO:teuthology.orchestra.run.smithi165.stdout:4159 write 0xe186ee8 thru 0xe18a651 (0x376a bytes) 2024-09-11T05:52:10.042 INFO:teuthology.orchestra.run.smithi165.stdout:4161 writesame 0xe5312e8 thru 0xe53d4f6 data_size 0x7(0xc20f bytes) 2024-09-11T05:52:10.045 INFO:teuthology.orchestra.run.smithi165.stdout:4162 write 0x90d1c9f thru 0x90df4d9 (0xd83b bytes) 2024-09-11T05:52:10.056 INFO:teuthology.orchestra.run.smithi165.stdout:4163 punch from 0xaf5ae1d to 0xaf60b39, (0x5d1c bytes) 2024-09-11T05:52:10.056 INFO:teuthology.orchestra.run.smithi165.stdout:4164 punch from 0xc709b06 to 0xc70bc71, (0x216b bytes) 2024-09-11T05:52:10.056 INFO:teuthology.orchestra.run.smithi165.stdout:4165 read 0xb4f1c38 thru 0xb4f660a (0x49d3 bytes) 2024-09-11T05:52:10.056 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.056 INFO:teuthology.orchestra.run.smithi165.stdout:4167 writesame 0xc431391 thru 0xc440fe8 data_size 0xbe(0xfc58 bytes) 2024-09-11T05:52:10.064 INFO:teuthology.orchestra.run.smithi165.stdout:4168 read 0xd168bc9 thru 0xd16fba9 (0x6fe1 bytes) 2024-09-11T05:52:10.065 INFO:teuthology.orchestra.run.smithi165.stdout:4169 punch from 0x51e2407 to 0x51e7ba3, (0x579c bytes) 2024-09-11T05:52:10.065 INFO:teuthology.orchestra.run.smithi165.stdout:4170 write 0x88dbce5 thru 0x88e024b (0x4567 bytes) 2024-09-11T05:52:10.068 INFO:teuthology.orchestra.run.smithi165.stdout:4172 punch from 0xbfd7ac0 to 0xbfe588d, (0xddcd bytes) 2024-09-11T05:52:10.083 INFO:teuthology.orchestra.run.smithi165.stdout:4174 trunc from 0xe53d4f7 to 0x45467db 2024-09-11T05:52:10.084 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734979868, 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734950459, 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734930402, 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734889015, 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734856532999999, 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:10.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:10.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734836477000002, 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:10.106 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:10.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734816975000001, 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:10.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:10.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734792165999998, 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:10.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734773016999998, 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.120 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:10.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:10.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734752202999999, 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.124 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:10.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.73473237, 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:10.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734712429999998, 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734686309000001, 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:10.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:10.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:10.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:10.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.734649725000001, 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.141 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:10.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:10.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.729861926000002, 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:10.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:10.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:10.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.729787178999999, 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.149 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.729714420000001, 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:10.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:10.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:10.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.7296935, 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:10.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.387727308999999, 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:10.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:10.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:10.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6792878980000001, 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:10.168 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:10.172 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:10.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:10.174 INFO:teuthology.orchestra.run.smithi165.stdout:4175 read 0x2c212a9 thru 0x2c28fb1 (0x7d09 bytes) 2024-09-11T05:52:10.174 INFO:teuthology.orchestra.run.smithi165.stdout:4176 punch from 0x23699a9 to 0x236b323, (0x197a bytes) 2024-09-11T05:52:10.175 INFO:teuthology.orchestra.run.smithi165.stdout:4177 trunc from 0x45467db to 0x25bc584 2024-09-11T05:52:10.175 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:10.172+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:10.176 INFO:teuthology.orchestra.run.smithi165.stdout:4180 write 0x2ab0f3a thru 0x2abd038 (0xc0ff bytes) 2024-09-11T05:52:10.180 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.195 INFO:teuthology.orchestra.run.smithi165.stdout:4184 writesame 0x4cb3b47 thru 0x4cbf704 data_size 0x7(0xbbbe bytes) 2024-09-11T05:52:10.195 INFO:teuthology.orchestra.run.smithi165.stdout:4186 write 0x39729e1 thru 0x3978834 (0x5e54 bytes) 2024-09-11T05:52:10.206 INFO:teuthology.orchestra.run.smithi165.stdout:4188 writesame 0x258d751 thru 0x258ed77 data_size 0x1(0x1627 bytes) 2024-09-11T05:52:10.227 INFO:teuthology.orchestra.run.smithi165.stdout:4189 write 0x76f8ea5 thru 0x7706707 (0xd863 bytes) 2024-09-11T05:52:10.248 INFO:teuthology.orchestra.run.smithi165.stdout:4190 trunc from 0x7706708 to 0xb4622d0 2024-09-11T05:52:10.258 INFO:teuthology.orchestra.run.smithi165.stdout:4192 read 0x3f9a3ff thru 0x3fa23e4 (0x7fe6 bytes) 2024-09-11T05:52:10.258 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.259 INFO:teuthology.orchestra.run.smithi165.stdout:4194 writesame 0x22cce0d thru 0x22cd81a data_size 0x16(0xa0e bytes) 2024-09-11T05:52:10.261 INFO:teuthology.orchestra.run.smithi165.stdout:4195 read 0x362d0d0 thru 0x363ad16 (0xdc47 bytes) 2024-09-11T05:52:10.275 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:10.271+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:10.277 INFO:teuthology.orchestra.run.smithi165.stdout:4197 write 0xc4e5c1a thru 0xc4e86a4 (0x2a8b bytes) 2024-09-11T05:52:10.288 INFO:teuthology.orchestra.run.smithi165.stdout:4198 write 0xa49595f thru 0xa4a2ce6 (0xd388 bytes) 2024-09-11T05:52:10.302 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.302 INFO:teuthology.orchestra.run.smithi165.stdout:4200 writesame 0x3995e56 thru 0x399c3a3 data_size 0x1(0x654e bytes) 2024-09-11T05:52:10.308 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:10.308 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:10.308 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:10.308 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:10.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:10.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:10.359 INFO:teuthology.orchestra.run.smithi165.stdout:4201 writesame 0xe8f1429 thru 0xe8f4d72 data_size 0x1(0x394a bytes) 2024-09-11T05:52:10.359 INFO:teuthology.orchestra.run.smithi165.stdout:4202 punch from 0x7622abb to 0x76255cc, (0x2b11 bytes) 2024-09-11T05:52:10.376 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:10.371+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:10.388 INFO:teuthology.orchestra.run.smithi165.stdout:4203 trunc from 0xe8f4d73 to 0x20bcf68 2024-09-11T05:52:10.423 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.424 INFO:teuthology.orchestra.run.smithi165.stdout:4205 read 0x2b76b6 thru 0x2bfd3d (0x8688 bytes) 2024-09-11T05:52:10.424 INFO:teuthology.orchestra.run.smithi165.stdout:4206 punch from 0xc15aab to 0xc1d828, (0x7d7d bytes) 2024-09-11T05:52:10.442 INFO:teuthology.orchestra.run.smithi165.stdout:4207 write 0x4f4ded6 thru 0x4f52e6b (0x4f96 bytes) 2024-09-11T05:52:10.453 INFO:teuthology.orchestra.run.smithi165.stdout:4208 read 0x12e3ff1 thru 0x12f3fc8 (0xffd8 bytes) 2024-09-11T05:52:10.477 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:10.471+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:10.477 INFO:teuthology.orchestra.run.smithi165.stdout:4209 writesame 0xa4f2edf thru 0xa4fa853 data_size 0x1f(0x7975 bytes) 2024-09-11T05:52:10.477 INFO:teuthology.orchestra.run.smithi165.stdout:4211 read 0x2bac89e thru 0x2bb163f (0x4da2 bytes) 2024-09-11T05:52:10.479 INFO:teuthology.orchestra.run.smithi165.stdout:4212 read 0x223889c thru 0x2247f72 (0xf6d7 bytes) 2024-09-11T05:52:10.480 INFO:teuthology.orchestra.run.smithi165.stdout:4213 read 0x97c47a9 thru 0x97d4071 (0xf8c9 bytes) 2024-09-11T05:52:10.482 INFO:teuthology.orchestra.run.smithi165.stdout:4214 trunc from 0xa4fa854 to 0x973e95d 2024-09-11T05:52:10.492 INFO:teuthology.orchestra.run.smithi165.stdout:4216 punch from 0x9602ce8 to 0x96031a9, (0x4c1 bytes) 2024-09-11T05:52:10.492 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.493 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:10.493 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:10.493 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:10.493 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:10.493 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:10.507 INFO:teuthology.orchestra.run.smithi165.stdout:4218 writesame 0xb57077d thru 0xb57df52 data_size 0x3(0xd7d6 bytes) 2024-09-11T05:52:10.507 INFO:teuthology.orchestra.run.smithi165.stdout:4219 read 0x2849cc4 thru 0x284bb6b (0x1ea8 bytes) 2024-09-11T05:52:10.507 INFO:teuthology.orchestra.run.smithi165.stdout:4220 read 0x2789112 thru 0x2793593 (0xa482 bytes) 2024-09-11T05:52:10.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:10.515 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:10.515 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:10.520 INFO:teuthology.orchestra.run.smithi165.stdout:4221 write 0xd664d6d thru 0xd66ddb8 (0x904c bytes) 2024-09-11T05:52:10.532 INFO:teuthology.orchestra.run.smithi165.stdout:4222 read 0xa440eaf thru 0xa44e897 (0xd9e9 bytes) 2024-09-11T05:52:10.532 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.532 INFO:teuthology.orchestra.run.smithi165.stdout:4224 read 0x892ec56 thru 0x89373b0 (0x875b bytes) 2024-09-11T05:52:10.532 INFO:teuthology.orchestra.run.smithi165.stdout:4226 write 0x222dc78 thru 0x2238ccb (0xb054 bytes) 2024-09-11T05:52:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:10.534 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:10.535 INFO:teuthology.orchestra.run.smithi165.stdout:4227 punch from 0x42aaf5b to 0x42ad117, (0x21bc bytes) 2024-09-11T05:52:10.542 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T05:52:10.550 INFO:teuthology.orchestra.run.smithi165.stdout:4228 writesame 0xe19497f thru 0xe1a01c5 data_size 0x25(0xb847 bytes) 2024-09-11T05:52:10.550 INFO:teuthology.orchestra.run.smithi165.stdout:4230 read 0xdd3316 thru 0xde2212 (0xeefd bytes) 2024-09-11T05:52:10.551 INFO:teuthology.orchestra.run.smithi165.stdout:4231 punch from 0x1c4a76a to 0x1c54876, (0xa10c bytes) 2024-09-11T05:52:10.554 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.554 INFO:teuthology.orchestra.run.smithi165.stdout:4234 read 0xdb92d59 thru 0xdba1fba (0xf262 bytes) 2024-09-11T05:52:10.554 INFO:teuthology.orchestra.run.smithi165.stdout:4235 read 0x1d0de6b thru 0x1d16219 (0x83af bytes) 2024-09-11T05:52:10.555 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:10.555 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:10.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:52:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:4237 read 0xa5b9e68 thru 0xa5c1bcd (0x7d66 bytes) 2024-09-11T05:52:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:4238 read 0xd9f2785 thru 0xd9ff3e5 (0xcc61 bytes) 2024-09-11T05:52:10.562 INFO:teuthology.orchestra.run.smithi165.stdout:4239 write 0x3981bb5 thru 0x39852e5 (0x3731 bytes) 2024-09-11T05:52:10.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-11T05:52:10.572 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:10.573 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:10.574 INFO:teuthology.orchestra.run.smithi165.stdout:4242 write 0xf41187 thru 0xf4b198 (0xa012 bytes) 2024-09-11T05:52:10.606 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:10.606 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:10.606 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:10.604+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:10.620 INFO:teuthology.orchestra.run.smithi165.stdout:4246 write 0x8f783d3 thru 0x8f787c8 (0x3f6 bytes) 2024-09-11T05:52:10.626 INFO:teuthology.orchestra.run.smithi165.stdout:4247 read 0x1cfe8c0 thru 0x1d0d4d1 (0xec12 bytes) 2024-09-11T05:52:10.628 INFO:teuthology.orchestra.run.smithi165.stdout:4248 write 0x7afc53 thru 0x7b248a (0x2838 bytes) 2024-09-11T05:52:10.628 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:10.628 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:10.633 INFO:teuthology.orchestra.run.smithi165.stdout:4249 read 0x4079f7c thru 0x4088403 (0xe488 bytes) 2024-09-11T05:52:10.633 INFO:teuthology.orchestra.run.smithi165.stdout:4250 punch from 0xdc6d6d9 to 0xdc6ea24, (0x134b bytes) 2024-09-11T05:52:10.633 INFO:teuthology.orchestra.run.smithi165.stdout:4251 punch from 0x8b32b31 to 0x8b39497, (0x6966 bytes) 2024-09-11T05:52:10.634 INFO:teuthology.orchestra.run.smithi165.stdout:4252 read 0x629f8aa thru 0x62a7adf (0x8236 bytes) 2024-09-11T05:52:10.634 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:10.634 INFO:teuthology.orchestra.run.smithi165.stdout:4254 writesame 0xb385d8f thru 0xb38a08e data_size 0x1(0x4300 bytes) 2024-09-11T05:52:10.648 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:10.648 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:10.649 INFO:teuthology.orchestra.run.smithi165.stdout:4255 read 0x427e6fd thru 0x42881b1 (0x9ab5 bytes) 2024-09-11T05:52:10.650 INFO:teuthology.orchestra.run.smithi165.stdout:4257 read 0x4367533 thru 0x4372825 (0xb2f3 bytes) 2024-09-11T05:52:10.651 INFO:teuthology.orchestra.run.smithi165.stdout:4258 read 0x925ce34 thru 0x9262fe6 (0x61b3 bytes) 2024-09-11T05:52:10.651 INFO:teuthology.orchestra.run.smithi165.stdout:4259 read 0x565db29 thru 0x5663bd4 (0x60ac bytes) 2024-09-11T05:52:10.651 INFO:teuthology.orchestra.run.smithi165.stdout:4260 writesame 0xcc4e37d thru 0xcc56f92 data_size 0x56(0x8c16 bytes) 2024-09-11T05:52:10.654 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:10.654 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:10.661 INFO:teuthology.orchestra.run.smithi165.stdout:4261 clone 28 order 25 su 524288 sc 15 2024-09-11T05:52:10.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:10.675 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:10.676 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:10.678 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:10.676+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:10.779 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:10.775+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:10.792 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:10.792 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:10.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:10.880 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:10.880+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:10.915 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:10.915 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:10.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:11.016 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:11.016+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:11.082 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:11.080+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:11.086 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.086 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:11.183 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:11.180+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:11.231 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.231 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:11.290 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:11.287+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:11.366 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.367 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:11.430 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:11.427+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:11.486 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:11.483+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:11.551 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.551 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.562 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T05:52:11.562 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:52:11.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:11.593 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T05:52:11.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:52:11.677 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:11.611+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:11.679 INFO:teuthology.orchestra.run.smithi165.stdout:leaving image image_client.0-clone27 intact 2024-09-11T05:52:11.720 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:11.720+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:11.790 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:11.788+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:11.812 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.812 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:11.891 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:11.888+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:11.893 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-09-11T05:52:11.895 INFO:teuthology.orchestra.run.smithi165.stdout:4262 write 0x15ca88f thru 0x15d0a46 (0x61b8 bytes) 2024-09-11T05:52:11.910 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:11.910 INFO:teuthology.orchestra.run.smithi165.stdout:4265 read 0xde78400 thru 0xde8386c (0xb46d bytes) 2024-09-11T05:52:11.910 INFO:teuthology.orchestra.run.smithi165.stdout:4266 write 0x69a9f5 thru 0x69da3f (0x304b bytes) 2024-09-11T05:52:11.913 INFO:teuthology.orchestra.run.smithi165.stdout:4269 trunc from 0xe1a01c6 to 0xe3be5e0 2024-09-11T05:52:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:4270 read 0x75fb393 thru 0x7604df1 (0x9a5f bytes) 2024-09-11T05:52:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:4271 punch from 0x90b48f8 to 0x90c278b, (0xde93 bytes) 2024-09-11T05:52:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:4272 write 0xe0d7586 thru 0xe0e6bd2 (0xf64d bytes) 2024-09-11T05:52:11.938 INFO:teuthology.orchestra.run.smithi165.stdout:4273 read 0x8b89bf5 thru 0x8b8f5b6 (0x59c2 bytes) 2024-09-11T05:52:11.939 INFO:teuthology.orchestra.run.smithi165.stdout:4275 punch from 0x3675fea to 0x367e173, (0x8189 bytes) 2024-09-11T05:52:11.939 INFO:teuthology.orchestra.run.smithi165.stdout:4277 read 0x29aea2a thru 0x29b1869 (0x2e40 bytes) 2024-09-11T05:52:11.949 INFO:teuthology.orchestra.run.smithi165.stdout:4278 write 0xe5528e4 thru 0xe55555b (0x2c78 bytes) 2024-09-11T05:52:11.956 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:11.956 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:11.968 INFO:teuthology.orchestra.run.smithi165.stdout:4279 writesame 0xcb007e5 thru 0xcb0b5ae data_size 0xf(0xadca bytes) 2024-09-11T05:52:11.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:11.988 INFO:teuthology.orchestra.run.smithi165.stdout:4281 read 0x3916e66 thru 0x391dab5 (0x6c50 bytes) 2024-09-11T05:52:11.989 INFO:teuthology.orchestra.run.smithi165.stdout:4283 punch from 0xa9e7e77 to 0xa9ee86c, (0x69f5 bytes) 2024-09-11T05:52:11.989 INFO:teuthology.orchestra.run.smithi165.stdout:4284 write 0x30f1b19 thru 0x30f708a (0x5572 bytes) 2024-09-11T05:52:12.004 INFO:teuthology.orchestra.run.smithi165.stdout:4285 punch from 0xb0c9aba to 0xb0ca827, (0xd6d bytes) 2024-09-11T05:52:12.004 INFO:teuthology.orchestra.run.smithi165.stdout:4286 writesame 0x3c6eeda thru 0x3c7ad23 data_size 0x45(0xbe4a bytes) 2024-09-11T05:52:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:52:12.021 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:52:12.022 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:12.019+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:12.024 INFO:teuthology.orchestra.run.smithi165.stdout:4287 punch from 0x99f8d98 to 0x99fa48f, (0x16f7 bytes) 2024-09-11T05:52:12.024 INFO:teuthology.orchestra.run.smithi165.stdout:4288 read 0xb620fd3 thru 0xb6295ad (0x85db bytes) 2024-09-11T05:52:12.025 INFO:teuthology.orchestra.run.smithi165.stdout:4289 punch from 0x6280a6b to 0x628f74a, (0xecdf bytes) 2024-09-11T05:52:12.025 INFO:teuthology.orchestra.run.smithi165.stdout:4290 write 0x1c190a6 thru 0x1c2017d (0x70d8 bytes) 2024-09-11T05:52:12.036 INFO:teuthology.orchestra.run.smithi165.stdout:4291 trunc from 0xe55555c to 0xc86e0d6 2024-09-11T05:52:12.041 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-11T05:52:11.568410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":784099948,"num_objects":1401,"num_object_clones":476,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":520,"num_read":153120,"num_read_kb":445628,"num_write":15027,"num_write_kb":242392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":26623623,"num_keys_recovered":60,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11149,"ondisk_log_size":11149,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":525784,"kb_used_data":227668,"kb_used_omap":12,"kb_used_meta":297907,"kb_avail":754448936,"statfs":{"total":773094113280,"available":772555710464,"internally_reserved":0,"allocated":233132032,"data_stored":299435953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":305057330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19052728,"num_objects":72,"num_object_clones":37,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":27,"num_read":6319,"num_read_kb":16008,"num_write":710,"num_write_kb":8497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":16676898,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.048030"},"pg_stats":[{"pgid":"3.1f","version":"74'276","reported_seq":1692,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542246+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:52:09.542246+0000","last_peered":"2024-09-11T05:52:09.542246+0000","last_clean":"2024-09-11T05:52:09.542246+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:52:09.542246+0000","last_undegraded":"2024-09-11T05:52:09.542246+0000","last_fullsized":"2024-09-11T05:52:09.542246+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.549968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":22870782,"num_objects":53,"num_object_clones":22,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":19,"num_read":1792,"num_read_kb":6133,"num_write":357,"num_write_kb":3784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"75'258","reported_seq":1802,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524872+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:52:08.524872+0000","last_peered":"2024-09-11T05:52:08.524872+0000","last_clean":"2024-09-11T05:52:08.524872+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:52:08.524872+0000","last_undegraded":"2024-09-11T05:52:08.524872+0000","last_fullsized":"2024-09-11T05:52:08.524872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:14:02.575844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":20546574,"num_objects":37,"num_object_clones":10,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":16,"num_read":2851,"num_read_kb":7328,"num_write":291,"num_write_kb":2723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"74'168","reported_seq":2987,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.568430+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:52:09.568430+0000","last_peered":"2024-09-11T05:52:09.568430+0000","last_clean":"2024-09-11T05:52:09.568430+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:52:09.568430+0000","last_undegraded":"2024-09-11T05:52:09.568430+0000","last_fullsized":"2024-09-11T05:52:09.568430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:45.474968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":21848884,"num_objects":38,"num_object_clones":12,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":13,"num_read":2723,"num_read_kb":13236,"num_write":282,"num_write_kb":6038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"77'324","reported_seq":2719,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.809944+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:52:09.809944+0000","last_peered":"2024-09-11T05:52:09.809944+0000","last_clean":"2024-09-11T05:52:09.809944+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:52:09.809944+0000","last_undegraded":"2024-09-11T05:52:09.809944+0000","last_fullsized":"2024-09-11T05:52:09.809944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:26.239138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":6416840,"num_objects":37,"num_object_clones":14,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":12,"num_read":3443,"num_read_kb":7242,"num_write":453,"num_write_kb":3361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'228","reported_seq":5857,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.809326+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:52:09.809326+0000","last_peered":"2024-09-11T05:52:09.809326+0000","last_clean":"2024-09-11T05:52:09.809326+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:52:09.809326+0000","last_undegraded":"2024-09-11T05:52:09.809326+0000","last_fullsized":"2024-09-11T05:52:09.809326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:13.434189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":20509215,"num_objects":38,"num_object_clones":13,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":13,"num_read":6128,"num_read_kb":20448,"num_write":273,"num_write_kb":7747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"71'262","reported_seq":7166,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.803901+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:52:09.803901+0000","last_peered":"2024-09-11T05:52:09.803901+0000","last_clean":"2024-09-11T05:52:09.803901+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:52:09.803901+0000","last_undegraded":"2024-09-11T05:52:09.803901+0000","last_fullsized":"2024-09-11T05:52:09.803901+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:34:54.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":27789460,"num_objects":50,"num_object_clones":16,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":22,"num_read":6952,"num_read_kb":12701,"num_write":365,"num_write_kb":11903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"77'483","reported_seq":3373,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.835892+0000","last_change":"2024-09-11T05:52:08.536765+0000","last_active":"2024-09-11T05:52:09.835892+0000","last_peered":"2024-09-11T05:52:09.835892+0000","last_clean":"2024-09-11T05:52:09.835892+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:52:09.835892+0000","last_undegraded":"2024-09-11T05:52:09.835892+0000","last_fullsized":"2024-09-11T05:52:09.835892+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:59.997012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":10754186,"num_objects":55,"num_object_clones":17,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":21,"num_read":4401,"num_read_kb":9313,"num_write":719,"num_write_kb":5522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"74'316","reported_seq":6518,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.525123+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:52:08.525123+0000","last_peered":"2024-09-11T05:52:08.525123+0000","last_clean":"2024-09-11T05:52:08.525123+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:52:08.525123+0000","last_undegraded":"2024-09-11T05:52:08.525123+0000","last_fullsized":"2024-09-11T05:52:08.525123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:18.219773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":29600218,"num_objects":52,"num_object_clones":19,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":15,"num_read":7459,"num_read_kb":25943,"num_write":377,"num_write_kb":9893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'266","reported_seq":2237,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542327+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:52:09.542327+0000","last_peered":"2024-09-11T05:52:09.542327+0000","last_clean":"2024-09-11T05:52:09.542327+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:52:09.542327+0000","last_undegraded":"2024-09-11T05:52:09.542327+0000","last_fullsized":"2024-09-11T05:52:09.542327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:52:52.653602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":6106764,"num_objects":43,"num_object_clones":16,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":18,"num_read":2685,"num_read_kb":5452,"num_write":360,"num_write_kb":5187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"77'190","reported_seq":8442,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.568343+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:52:09.568343+0000","last_peered":"2024-09-11T05:52:09.568343+0000","last_clean":"2024-09-11T05:52:09.568343+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:52:09.568343+0000","last_undegraded":"2024-09-11T05:52:09.568343+0000","last_fullsized":"2024-09-11T05:52:09.568343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:31.284002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":29916950,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":19,"num_read":8153,"num_read_kb":22857,"num_write":259,"num_write_kb":12216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"77'315","reported_seq":6849,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.873482+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:52:09.873482+0000","last_peered":"2024-09-11T05:52:09.873482+0000","last_clean":"2024-09-11T05:52:09.873482+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:52:09.873482+0000","last_undegraded":"2024-09-11T05:52:09.873482+0000","last_fullsized":"2024-09-11T05:52:09.873482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:48:39.140705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":28320474,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":12,"num_read":7499,"num_read_kb":9773,"num_write":323,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"77'606","reported_seq":4697,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.700215+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:52:09.700215+0000","last_peered":"2024-09-11T05:52:09.700215+0000","last_clean":"2024-09-11T05:52:09.700215+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:52:09.700215+0000","last_undegraded":"2024-09-11T05:52:09.700215+0000","last_fullsized":"2024-09-11T05:52:09.700215+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:49.872658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":38063081,"num_objects":60,"num_object_clones":24,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":18,"num_read":6458,"num_read_kb":19785,"num_write":852,"num_write_kb":11512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557642+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:52:09.557642+0000","last_peered":"2024-09-11T05:52:09.557642+0000","last_clean":"2024-09-11T05:52:09.557642+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:52:09.557642+0000","last_undegraded":"2024-09-11T05:52:09.557642+0000","last_fullsized":"2024-09-11T05:52:09.557642+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"77'517","reported_seq":3899,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542226+0000","last_change":"2024-09-11T05:52:08.530537+0000","last_active":"2024-09-11T05:52:09.542226+0000","last_peered":"2024-09-11T05:52:09.542226+0000","last_clean":"2024-09-11T05:52:09.542226+0000","last_became_active":"2024-09-11T05:52:08.529499+0000","last_became_peered":"2024-09-11T05:52:08.529499+0000","last_unstale":"2024-09-11T05:52:09.542226+0000","last_undegraded":"2024-09-11T05:52:09.542226+0000","last_fullsized":"2024-09-11T05:52:09.542226+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:48.318545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":17401288,"num_objects":49,"num_object_clones":17,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":20,"num_read":4618,"num_read_kb":10568,"num_write":741,"num_write_kb":5301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4454693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.537731+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:52:09.537731+0000","last_peered":"2024-09-11T05:52:09.537731+0000","last_clean":"2024-09-11T05:52:09.537731+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:52:09.537731+0000","last_undegraded":"2024-09-11T05:52:09.537731+0000","last_fullsized":"2024-09-11T05:52:09.537731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"74'335","reported_seq":2012,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524579+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:52:08.524579+0000","last_peered":"2024-09-11T05:52:08.524579+0000","last_clean":"2024-09-11T05:52:08.524579+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:52:08.524579+0000","last_undegraded":"2024-09-11T05:52:08.524579+0000","last_fullsized":"2024-09-11T05:52:08.524579+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:01.900974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":10308920,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":14,"num_read":2525,"num_read_kb":7930,"num_write":516,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557497+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:52:09.557497+0000","last_peered":"2024-09-11T05:52:09.557497+0000","last_clean":"2024-09-11T05:52:09.557497+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:52:09.557497+0000","last_undegraded":"2024-09-11T05:52:09.557497+0000","last_fullsized":"2024-09-11T05:52:09.557497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"77'215","reported_seq":4712,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.683008+0000","last_change":"2024-09-11T05:52:08.530178+0000","last_active":"2024-09-11T05:52:09.683008+0000","last_peered":"2024-09-11T05:52:09.683008+0000","last_clean":"2024-09-11T05:52:09.683008+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:52:09.683008+0000","last_undegraded":"2024-09-11T05:52:09.683008+0000","last_fullsized":"2024-09-11T05:52:09.683008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:27:20.959367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":19792889,"num_objects":23,"num_object_clones":10,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":8,"num_read":4217,"num_read_kb":22070,"num_write":230,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.538008+0000","last_change":"2024-09-11T05:52:05.512835+0000","last_active":"2024-09-11T05:52:09.538008+0000","last_peered":"2024-09-11T05:52:09.538008+0000","last_clean":"2024-09-11T05:52:09.538008+0000","last_became_active":"2024-09-11T05:52:05.512365+0000","last_became_peered":"2024-09-11T05:52:05.512365+0000","last_unstale":"2024-09-11T05:52:09.538008+0000","last_undegraded":"2024-09-11T05:52:09.538008+0000","last_fullsized":"2024-09-11T05:52:09.538008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:58.934662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'292","reported_seq":3989,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.717908+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:52:09.717908+0000","last_peered":"2024-09-11T05:52:09.717908+0000","last_clean":"2024-09-11T05:52:09.717908+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:52:09.717908+0000","last_undegraded":"2024-09-11T05:52:09.717908+0000","last_fullsized":"2024-09-11T05:52:09.717908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:54:17.463045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":50243907,"num_objects":42,"num_object_clones":15,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":16,"num_read":4870,"num_read_kb":17910,"num_write":328,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"77'445","reported_seq":2585,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.873362+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:52:09.873362+0000","last_peered":"2024-09-11T05:52:09.873362+0000","last_clean":"2024-09-11T05:52:09.873362+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:52:09.873362+0000","last_undegraded":"2024-09-11T05:52:09.873362+0000","last_fullsized":"2024-09-11T05:52:09.873362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:32.356132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":11131449,"num_objects":41,"num_object_clones":13,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":15,"num_read":3768,"num_read_kb":8237,"num_write":682,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":131,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.567937+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:52:09.567937+0000","last_peered":"2024-09-11T05:52:09.567937+0000","last_clean":"2024-09-11T05:52:09.567937+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:52:09.567937+0000","last_undegraded":"2024-09-11T05:52:09.567937+0000","last_fullsized":"2024-09-11T05:52:09.567937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"77'434","reported_seq":2475,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.813643+0000","last_change":"2024-09-11T05:52:08.529630+0000","last_active":"2024-09-11T05:52:09.813643+0000","last_peered":"2024-09-11T05:52:09.813643+0000","last_clean":"2024-09-11T05:52:09.813643+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:52:09.813643+0000","last_undegraded":"2024-09-11T05:52:09.813643+0000","last_fullsized":"2024-09-11T05:52:09.813643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:17:39.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":18211408,"num_objects":46,"num_object_clones":14,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":17,"num_read":3609,"num_read_kb":9208,"num_write":526,"num_write_kb":4821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.634343+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:52:09.634343+0000","last_peered":"2024-09-11T05:52:09.634343+0000","last_clean":"2024-09-11T05:52:09.634343+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:52:09.634343+0000","last_undegraded":"2024-09-11T05:52:09.634343+0000","last_fullsized":"2024-09-11T05:52:09.634343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"77'475","reported_seq":4565,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.737806+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:52:09.737806+0000","last_peered":"2024-09-11T05:52:09.737806+0000","last_clean":"2024-09-11T05:52:09.737806+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:52:09.737806+0000","last_undegraded":"2024-09-11T05:52:09.737806+0000","last_fullsized":"2024-09-11T05:52:09.737806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:45:29.452238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":28942436,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":22,"num_read":7114,"num_read_kb":17045,"num_write":601,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.556838+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:52:09.556838+0000","last_peered":"2024-09-11T05:52:09.556838+0000","last_clean":"2024-09-11T05:52:09.556838+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:52:09.556838+0000","last_undegraded":"2024-09-11T05:52:09.556838+0000","last_fullsized":"2024-09-11T05:52:09.556838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"75'321","reported_seq":6052,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524902+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:52:08.524902+0000","last_peered":"2024-09-11T05:52:08.524902+0000","last_clean":"2024-09-11T05:52:08.524902+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:52:08.524902+0000","last_undegraded":"2024-09-11T05:52:08.524902+0000","last_fullsized":"2024-09-11T05:52:08.524902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:32:06.752512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":38781364,"num_objects":47,"num_object_clones":17,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":16,"num_read":6194,"num_read_kb":16690,"num_write":458,"num_write_kb":11923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":130,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.567937+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:52:09.567937+0000","last_peered":"2024-09-11T05:52:09.567937+0000","last_clean":"2024-09-11T05:52:09.567937+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:52:09.567937+0000","last_undegraded":"2024-09-11T05:52:09.567937+0000","last_fullsized":"2024-09-11T05:52:09.567937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":279,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.556926+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:52:09.556926+0000","last_peered":"2024-09-11T05:52:09.556926+0000","last_clean":"2024-09-11T05:52:09.556926+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:52:09.556926+0000","last_undegraded":"2024-09-11T05:52:09.556926+0000","last_fullsized":"2024-09-11T05:52:09.556926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"76'256","reported_seq":5071,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.534192+0000","last_change":"2024-09-11T05:52:08.528671+0000","last_active":"2024-09-11T05:52:08.534192+0000","last_peered":"2024-09-11T05:52:08.534192+0000","last_clean":"2024-09-11T05:52:08.534192+0000","last_became_active":"2024-09-11T05:52:08.527718+0000","last_became_peered":"2024-09-11T05:52:08.527718+0000","last_unstale":"2024-09-11T05:52:08.534192+0000","last_undegraded":"2024-09-11T05:52:08.534192+0000","last_fullsized":"2024-09-11T05:52:08.534192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:04:17.998135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":31506581,"num_objects":43,"num_object_clones":16,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":4520,"num_read_kb":14610,"num_write":308,"num_write_kb":16330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"74'189","reported_seq":2711,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.548263+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:52:09.548263+0000","last_peered":"2024-09-11T05:52:09.548263+0000","last_clean":"2024-09-11T05:52:09.548263+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:52:09.548263+0000","last_undegraded":"2024-09-11T05:52:09.548263+0000","last_fullsized":"2024-09-11T05:52:09.548263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:47.441213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":8665108,"num_objects":45,"num_object_clones":15,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":19,"num_read":2465,"num_read_kb":10301,"num_write":170,"num_write_kb":2686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"77'380","reported_seq":4262,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.842090+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:52:09.842090+0000","last_peered":"2024-09-11T05:52:09.842090+0000","last_clean":"2024-09-11T05:52:09.842090+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:52:09.842090+0000","last_undegraded":"2024-09-11T05:52:09.842090+0000","last_fullsized":"2024-09-11T05:52:09.842090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:05:27.672754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153508,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":15,"num_read":5618,"num_read_kb":13156,"num_write":469,"num_write_kb":9741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"74'238","reported_seq":2180,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.525039+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:52:08.525039+0000","last_peered":"2024-09-11T05:52:08.525039+0000","last_clean":"2024-09-11T05:52:08.525039+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:52:08.525039+0000","last_undegraded":"2024-09-11T05:52:08.525039+0000","last_fullsized":"2024-09-11T05:52:08.525039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:53:22.714654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":26324894,"num_objects":46,"num_object_clones":16,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":18,"num_read":2233,"num_read_kb":14444,"num_write":279,"num_write_kb":7816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"77'272","reported_seq":4702,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.841547+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:52:09.841547+0000","last_peered":"2024-09-11T05:52:09.841547+0000","last_clean":"2024-09-11T05:52:09.841547+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:52:09.841547+0000","last_undegraded":"2024-09-11T05:52:09.841547+0000","last_fullsized":"2024-09-11T05:52:09.841547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:21.993332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":10083137,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":13,"num_read":4713,"num_read_kb":7847,"num_write":366,"num_write_kb":7093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"74'238","reported_seq":3907,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557321+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:52:09.557321+0000","last_peered":"2024-09-11T05:52:09.557321+0000","last_clean":"2024-09-11T05:52:09.557321+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:52:09.557321+0000","last_undegraded":"2024-09-11T05:52:09.557321+0000","last_fullsized":"2024-09-11T05:52:09.557321+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:02:20.555045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":15909655,"num_objects":41,"num_object_clones":14,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":16,"num_read":3952,"num_read_kb":15169,"num_write":287,"num_write_kb":4403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"74'717","reported_seq":7769,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.781078+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:52:09.781078+0000","last_peered":"2024-09-11T05:52:09.781078+0000","last_clean":"2024-09-11T05:52:09.781078+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:52:09.781078+0000","last_undegraded":"2024-09-11T05:52:09.781078+0000","last_fullsized":"2024-09-11T05:52:09.781078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:00:50.860848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":44084798,"num_objects":44,"num_object_clones":13,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":13,"num_read":9776,"num_read_kb":27990,"num_write":1023,"num_write_kb":14173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"74'351","reported_seq":2367,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.773585+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:52:09.773585+0000","last_peered":"2024-09-11T05:52:09.773585+0000","last_clean":"2024-09-11T05:52:09.773585+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:52:09.773585+0000","last_undegraded":"2024-09-11T05:52:09.773585+0000","last_fullsized":"2024-09-11T05:52:09.773585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:46:47.228514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":42069883,"num_objects":43,"num_object_clones":14,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":18,"num_read":2931,"num_read_kb":13109,"num_write":546,"num_write_kb":8021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"74'664","reported_seq":2336,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.794895+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:52:09.794895+0000","last_peered":"2024-09-11T05:52:09.794895+0000","last_clean":"2024-09-11T05:52:09.794895+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:52:09.794895+0000","last_undegraded":"2024-09-11T05:52:09.794895+0000","last_fullsized":"2024-09-11T05:52:09.794895+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":664,"log_dups_size":0,"ondisk_log_size":664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:10.161691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861883,"num_objects":47,"num_object_clones":18,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":18,"num_read":4363,"num_read_kb":20603,"num_write":1077,"num_write_kb":9932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"77'262","reported_seq":3680,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.886405+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:52:09.886405+0000","last_peered":"2024-09-11T05:52:09.886405+0000","last_clean":"2024-09-11T05:52:09.886405+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:52:09.886405+0000","last_undegraded":"2024-09-11T05:52:09.886405+0000","last_fullsized":"2024-09-11T05:52:09.886405+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:05:30.808687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":21867925,"num_objects":54,"num_object_clones":21,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":19,"num_read":4012,"num_read_kb":10375,"num_write":316,"num_write_kb":4741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"77'487","reported_seq":2547,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.859796+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:52:09.859796+0000","last_peered":"2024-09-11T05:52:09.859796+0000","last_clean":"2024-09-11T05:52:09.859796+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:52:09.859796+0000","last_undegraded":"2024-09-11T05:52:09.859796+0000","last_fullsized":"2024-09-11T05:52:09.859796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:55:37.678628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":9355868,"num_objects":38,"num_object_clones":13,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":15,"num_read":3826,"num_read_kb":12034,"num_write":694,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"74'249","reported_seq":6989,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557045+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:52:09.557045+0000","last_peered":"2024-09-11T05:52:09.557045+0000","last_clean":"2024-09-11T05:52:09.557045+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:52:09.557045+0000","last_undegraded":"2024-09-11T05:52:09.557045+0000","last_fullsized":"2024-09-11T05:52:09.557045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:32.394874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":43069232,"num_objects":50,"num_object_clones":18,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":16,"num_read":7136,"num_read_kb":16021,"num_write":338,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":783509561,"num_objects":1397,"num_object_clones":476,"num_object_copies":2794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":520,"num_read":153004,"num_read_kb":445528,"num_write":14866,"num_write_kb":239974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":26623623,"num_keys_recovered":60,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":224972800,"data_stored":295144875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11029,"ondisk_log_size":11029,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64124,"kb_used_data":37092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94307716,"statfs":{"total":96636764160,"available":96571101184,"internally_reserved":0,"allocated":37982208,"data_stored":49282787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50904,"kb_used_data":15556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320936,"statfs":{"total":96636764160,"available":96584638464,"internally_reserved":0,"allocated":15929344,"data_stored":18054539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64516,"kb_used_data":25008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94307324,"statfs":{"total":96636764160,"available":96570699776,"internally_reserved":0,"allocated":25608192,"data_stored":35230570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97872,"kb_used_data":52796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94273968,"statfs":{"total":96636764160,"available":96536543232,"internally_reserved":0,"allocated":54063104,"data_stored":73124282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70488,"kb_used_data":30980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94301352,"statfs":{"total":96636764160,"available":96564584448,"internally_reserved":0,"allocated":31723520,"data_stored":39186001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39008,"kb_used_data":7812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332832,"statfs":{"total":96636764160,"available":96596819968,"internally_reserved":0,"allocated":7999488,"data_stored":9329092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56680,"kb_used_data":21324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315160,"statfs":{"total":96636764160,"available":96578723840,"internally_reserved":0,"allocated":21835776,"data_stored":27380030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82192,"kb_used_data":37100,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94289648,"statfs":{"total":96636764160,"available":96552599552,"internally_reserved":0,"allocated":37990400,"data_stored":47848652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36532224,"data_stored":46869388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20955136,"data_stored":26993974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7110656,"data_stored":8926637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":38783546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53215232,"data_stored":72743702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24735744,"data_stored":34844496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15052800,"data_stored":17660391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36536320,"data_stored":48322741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:52:12.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:52:12.092 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:12.087+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:12.093 INFO:teuthology.orchestra.run.smithi165.stdout:4292 trunc from 0xc86e0d6 to 0x31de815 2024-09-11T05:52:12.107 INFO:teuthology.orchestra.run.smithi165.stdout:4293 read 0x2440f29 thru 0x2448704 (0x77dc bytes) 2024-09-11T05:52:12.107 INFO:teuthology.orchestra.run.smithi165.stdout:4294 punch from 0x22c853f to 0x22d583c, (0xd2fd bytes) 2024-09-11T05:52:12.123 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.124 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.126 INFO:teuthology.orchestra.run.smithi165.stdout:4295 write 0xe7f71a3 thru 0xe800397 (0x91f5 bytes) 2024-09-11T05:52:12.129 INFO:teuthology.orchestra.run.smithi165.stdout:4296 punch from 0x465c241 to 0x466acfc, (0xeabb bytes) 2024-09-11T05:52:12.129 INFO:teuthology.orchestra.run.smithi165.stdout:4297 read 0xa18844 thru 0xa1df27 (0x56e4 bytes) 2024-09-11T05:52:12.129 INFO:teuthology.orchestra.run.smithi165.stdout:4298 write 0x9847329 thru 0x9853962 (0xc63a bytes) 2024-09-11T05:52:12.131 INFO:teuthology.orchestra.run.smithi165.stdout:4299 punch from 0xd337f73 to 0xd33df66, (0x5ff3 bytes) 2024-09-11T05:52:12.131 INFO:teuthology.orchestra.run.smithi165.stdout:4300 write 0x2f83be1 thru 0x2f87fcb (0x43eb bytes) 2024-09-11T05:52:12.133 INFO:teuthology.orchestra.run.smithi165.stdout:4302 punch from 0x8b42a9a to 0x8b5121a, (0xe780 bytes) 2024-09-11T05:52:12.134 INFO:teuthology.orchestra.run.smithi165.stdout:4303 read 0xbb51c28 thru 0xbb58226 (0x65ff bytes) 2024-09-11T05:52:12.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:12.140 INFO:teuthology.orchestra.run.smithi165.stdout:4305 write 0xeb70b56 thru 0xeb7441f (0x38ca bytes) 2024-09-11T05:52:12.142 INFO:teuthology.orchestra.run.smithi165.stdout:4306 write 0x141c6b0 thru 0x1420b62 (0x44b3 bytes) 2024-09-11T05:52:12.143 INFO:teuthology.orchestra.run.smithi165.stdout:4307 write 0xac3baff thru 0xac49992 (0xde94 bytes) 2024-09-11T05:52:12.145 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.145 INFO:teuthology.orchestra.run.smithi165.stdout:4310 write 0x227f96 thru 0x22e6e6 (0x6751 bytes) 2024-09-11T05:52:12.148 INFO:teuthology.orchestra.run.smithi165.stdout:4311 write 0x65a0877 thru 0x65af5ac (0xed36 bytes) 2024-09-11T05:52:12.161 INFO:teuthology.orchestra.run.smithi165.stdout:4312 trunc from 0xeb74420 to 0x5a1fb12 2024-09-11T05:52:12.173 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.173 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.173 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.190 INFO:teuthology.orchestra.run.smithi165.stdout:4319 write 0xcdacdda thru 0xcdb405e (0x7285 bytes) 2024-09-11T05:52:12.192 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:12.192+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:12.203 INFO:teuthology.orchestra.run.smithi165.stdout:4321 writesame 0xe4e54b5 thru 0xe4f145e data_size 0x1(0xbfaa bytes) 2024-09-11T05:52:12.205 INFO:teuthology.orchestra.run.smithi165.stdout:4322 write 0xd6d61d4 thru 0xd6df337 (0x9164 bytes) 2024-09-11T05:52:12.207 INFO:teuthology.orchestra.run.smithi165.stdout:4324 read 0xd79e762 thru 0xd79f095 (0x934 bytes) 2024-09-11T05:52:12.215 INFO:teuthology.orchestra.run.smithi165.stdout:4325 write 0xe76226e thru 0xe766793 (0x4526 bytes) 2024-09-11T05:52:12.232 INFO:teuthology.orchestra.run.smithi165.stdout:4326 trunc from 0xe766794 to 0x3e1871c 2024-09-11T05:52:12.246 INFO:teuthology.orchestra.run.smithi165.stdout:4327 read 0x2822fa thru 0x2879b4 (0x56bb bytes) 2024-09-11T05:52:12.261 INFO:teuthology.orchestra.run.smithi165.stdout:4328 write 0x9ca887c thru 0x9cb8676 (0xfdfb bytes) 2024-09-11T05:52:12.263 INFO:teuthology.orchestra.run.smithi165.stdout:4329 writesame 0x650f07 thru 0x656453 data_size 0x1d(0x554d bytes) 2024-09-11T05:52:12.267 INFO:teuthology.orchestra.run.smithi165.stdout:4330 punch from 0xe4b658 to 0xe575a5, (0xbf4d bytes) 2024-09-11T05:52:12.267 INFO:teuthology.orchestra.run.smithi165.stdout:4331 read 0x1f57abd thru 0x1f58b4b (0x108f bytes) 2024-09-11T05:52:12.267 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.277 INFO:teuthology.orchestra.run.smithi165.stdout:4334 write 0xa7c79ed thru 0xa7cd0ad (0x56c1 bytes) 2024-09-11T05:52:12.279 INFO:teuthology.orchestra.run.smithi165.stdout:4335 write 0x97b78d2 thru 0x97bf9a6 (0x80d5 bytes) 2024-09-11T05:52:12.282 INFO:teuthology.orchestra.run.smithi165.stdout:4336 writesame 0x49d65f4 thru 0x49e197a data_size 0x1(0xb387 bytes) 2024-09-11T05:52:12.293 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:12.292+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:12.315 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.316 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:12.334 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.334 INFO:teuthology.orchestra.run.smithi165.stdout:4338 read 0x6816416 thru 0x6825b51 (0xf73c bytes) 2024-09-11T05:52:12.335 INFO:teuthology.orchestra.run.smithi165.stdout:4339 write 0x4cec494 thru 0x4cf9aa2 (0xd60f bytes) 2024-09-11T05:52:12.354 INFO:teuthology.orchestra.run.smithi165.stdout:4340 trunc from 0xa7cd0ae to 0x2afe54f 2024-09-11T05:52:12.370 INFO:teuthology.orchestra.run.smithi165.stdout:4341 punch from 0x132d8b to 0x13d350, (0xa5c5 bytes) 2024-09-11T05:52:12.387 INFO:teuthology.orchestra.run.smithi165.stdout:4343 write 0x6572330 thru 0x6580ac6 (0xe797 bytes) 2024-09-11T05:52:12.394 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:12.391+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:52:12.404 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:52:12.405 INFO:teuthology.orchestra.run.smithi165.stdout:4345 trunc from 0x6580ac7 to 0xc2e08b7 2024-09-11T05:52:12.410 INFO:teuthology.orchestra.run.smithi165.stdout:4347 writesame 0x69ba135 thru 0x69c7a77 data_size 0x1(0xd943 bytes) 2024-09-11T05:52:12.421 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-11T05:52:11.568410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":784099948,"num_objects":1401,"num_object_clones":476,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":520,"num_read":153120,"num_read_kb":445628,"num_write":15027,"num_write_kb":242392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":26623623,"num_keys_recovered":60,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11149,"ondisk_log_size":11149,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":525784,"kb_used_data":227668,"kb_used_omap":12,"kb_used_meta":297907,"kb_avail":754448936,"statfs":{"total":773094113280,"available":772555710464,"internally_reserved":0,"allocated":233132032,"data_stored":299435953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":305057330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19052728,"num_objects":72,"num_object_clones":37,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":27,"num_read":6319,"num_read_kb":16008,"num_write":710,"num_write_kb":8497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":16676898,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.048030"},"pg_stats":[{"pgid":"3.1f","version":"74'276","reported_seq":1692,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542246+0000","last_change":"2024-09-11T05:51:07.517861+0000","last_active":"2024-09-11T05:52:09.542246+0000","last_peered":"2024-09-11T05:52:09.542246+0000","last_clean":"2024-09-11T05:52:09.542246+0000","last_became_active":"2024-09-11T05:51:05.519792+0000","last_became_peered":"2024-09-11T05:51:05.519792+0000","last_unstale":"2024-09-11T05:52:09.542246+0000","last_undegraded":"2024-09-11T05:52:09.542246+0000","last_fullsized":"2024-09-11T05:52:09.542246+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.549968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":22870782,"num_objects":53,"num_object_clones":22,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":19,"num_read":1792,"num_read_kb":6133,"num_write":357,"num_write_kb":3784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"75'258","reported_seq":1802,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524872+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:52:08.524872+0000","last_peered":"2024-09-11T05:52:08.524872+0000","last_clean":"2024-09-11T05:52:08.524872+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:52:08.524872+0000","last_undegraded":"2024-09-11T05:52:08.524872+0000","last_fullsized":"2024-09-11T05:52:08.524872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:14:02.575844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":20546574,"num_objects":37,"num_object_clones":10,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":16,"num_read":2851,"num_read_kb":7328,"num_write":291,"num_write_kb":2723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"74'168","reported_seq":2987,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.568430+0000","last_change":"2024-09-11T05:51:07.519114+0000","last_active":"2024-09-11T05:52:09.568430+0000","last_peered":"2024-09-11T05:52:09.568430+0000","last_clean":"2024-09-11T05:52:09.568430+0000","last_became_active":"2024-09-11T05:51:05.521417+0000","last_became_peered":"2024-09-11T05:51:05.521417+0000","last_unstale":"2024-09-11T05:52:09.568430+0000","last_undegraded":"2024-09-11T05:52:09.568430+0000","last_fullsized":"2024-09-11T05:52:09.568430+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:45.474968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":21848884,"num_objects":38,"num_object_clones":12,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":13,"num_read":2723,"num_read_kb":13236,"num_write":282,"num_write_kb":6038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"77'324","reported_seq":2719,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.809944+0000","last_change":"2024-09-11T05:51:07.519534+0000","last_active":"2024-09-11T05:52:09.809944+0000","last_peered":"2024-09-11T05:52:09.809944+0000","last_clean":"2024-09-11T05:52:09.809944+0000","last_became_active":"2024-09-11T05:51:05.521550+0000","last_became_peered":"2024-09-11T05:51:05.521550+0000","last_unstale":"2024-09-11T05:52:09.809944+0000","last_undegraded":"2024-09-11T05:52:09.809944+0000","last_fullsized":"2024-09-11T05:52:09.809944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:26.239138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":6416840,"num_objects":37,"num_object_clones":14,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":12,"num_read":3443,"num_read_kb":7242,"num_write":453,"num_write_kb":3361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'228","reported_seq":5857,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.809326+0000","last_change":"2024-09-11T05:51:07.519552+0000","last_active":"2024-09-11T05:52:09.809326+0000","last_peered":"2024-09-11T05:52:09.809326+0000","last_clean":"2024-09-11T05:52:09.809326+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:52:09.809326+0000","last_undegraded":"2024-09-11T05:52:09.809326+0000","last_fullsized":"2024-09-11T05:52:09.809326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:13.434189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":20509215,"num_objects":38,"num_object_clones":13,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":13,"num_read":6128,"num_read_kb":20448,"num_write":273,"num_write_kb":7747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"71'262","reported_seq":7166,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.803901+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:52:09.803901+0000","last_peered":"2024-09-11T05:52:09.803901+0000","last_clean":"2024-09-11T05:52:09.803901+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:52:09.803901+0000","last_undegraded":"2024-09-11T05:52:09.803901+0000","last_fullsized":"2024-09-11T05:52:09.803901+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:34:54.992843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":27789460,"num_objects":50,"num_object_clones":16,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":22,"num_read":6952,"num_read_kb":12701,"num_write":365,"num_write_kb":11903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"77'483","reported_seq":3373,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.835892+0000","last_change":"2024-09-11T05:52:08.536765+0000","last_active":"2024-09-11T05:52:09.835892+0000","last_peered":"2024-09-11T05:52:09.835892+0000","last_clean":"2024-09-11T05:52:09.835892+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:52:09.835892+0000","last_undegraded":"2024-09-11T05:52:09.835892+0000","last_fullsized":"2024-09-11T05:52:09.835892+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:59.997012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":10754186,"num_objects":55,"num_object_clones":17,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":21,"num_read":4401,"num_read_kb":9313,"num_write":719,"num_write_kb":5522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"74'316","reported_seq":6518,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.525123+0000","last_change":"2024-09-11T05:51:07.507472+0000","last_active":"2024-09-11T05:52:08.525123+0000","last_peered":"2024-09-11T05:52:08.525123+0000","last_clean":"2024-09-11T05:52:08.525123+0000","last_became_active":"2024-09-11T05:51:05.507595+0000","last_became_peered":"2024-09-11T05:51:05.507595+0000","last_unstale":"2024-09-11T05:52:08.525123+0000","last_undegraded":"2024-09-11T05:52:08.525123+0000","last_fullsized":"2024-09-11T05:52:08.525123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:18.219773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":29600218,"num_objects":52,"num_object_clones":19,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":15,"num_read":7459,"num_read_kb":25943,"num_write":377,"num_write_kb":9893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'266","reported_seq":2237,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542327+0000","last_change":"2024-09-11T05:51:07.519671+0000","last_active":"2024-09-11T05:52:09.542327+0000","last_peered":"2024-09-11T05:52:09.542327+0000","last_clean":"2024-09-11T05:52:09.542327+0000","last_became_active":"2024-09-11T05:51:05.519636+0000","last_became_peered":"2024-09-11T05:51:05.519636+0000","last_unstale":"2024-09-11T05:52:09.542327+0000","last_undegraded":"2024-09-11T05:52:09.542327+0000","last_fullsized":"2024-09-11T05:52:09.542327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:52:52.653602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":6106764,"num_objects":43,"num_object_clones":16,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":18,"num_read":2685,"num_read_kb":5452,"num_write":360,"num_write_kb":5187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"77'190","reported_seq":8442,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.568343+0000","last_change":"2024-09-11T05:51:07.518839+0000","last_active":"2024-09-11T05:52:09.568343+0000","last_peered":"2024-09-11T05:52:09.568343+0000","last_clean":"2024-09-11T05:52:09.568343+0000","last_became_active":"2024-09-11T05:51:05.518539+0000","last_became_peered":"2024-09-11T05:51:05.518539+0000","last_unstale":"2024-09-11T05:52:09.568343+0000","last_undegraded":"2024-09-11T05:52:09.568343+0000","last_fullsized":"2024-09-11T05:52:09.568343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:31.284002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":29916950,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":19,"num_read":8153,"num_read_kb":22857,"num_write":259,"num_write_kb":12216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"77'315","reported_seq":6849,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.873482+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:52:09.873482+0000","last_peered":"2024-09-11T05:52:09.873482+0000","last_clean":"2024-09-11T05:52:09.873482+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:52:09.873482+0000","last_undegraded":"2024-09-11T05:52:09.873482+0000","last_fullsized":"2024-09-11T05:52:09.873482+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:48:39.140705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":28320474,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":12,"num_read":7499,"num_read_kb":9773,"num_write":323,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"77'606","reported_seq":4697,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.700215+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:52:09.700215+0000","last_peered":"2024-09-11T05:52:09.700215+0000","last_clean":"2024-09-11T05:52:09.700215+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:52:09.700215+0000","last_undegraded":"2024-09-11T05:52:09.700215+0000","last_fullsized":"2024-09-11T05:52:09.700215+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:49.872658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":38063081,"num_objects":60,"num_object_clones":24,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":18,"num_read":6458,"num_read_kb":19785,"num_write":852,"num_write_kb":11512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557642+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:52:09.557642+0000","last_peered":"2024-09-11T05:52:09.557642+0000","last_clean":"2024-09-11T05:52:09.557642+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:52:09.557642+0000","last_undegraded":"2024-09-11T05:52:09.557642+0000","last_fullsized":"2024-09-11T05:52:09.557642+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"77'517","reported_seq":3899,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.542226+0000","last_change":"2024-09-11T05:52:08.530537+0000","last_active":"2024-09-11T05:52:09.542226+0000","last_peered":"2024-09-11T05:52:09.542226+0000","last_clean":"2024-09-11T05:52:09.542226+0000","last_became_active":"2024-09-11T05:52:08.529499+0000","last_became_peered":"2024-09-11T05:52:08.529499+0000","last_unstale":"2024-09-11T05:52:09.542226+0000","last_undegraded":"2024-09-11T05:52:09.542226+0000","last_fullsized":"2024-09-11T05:52:09.542226+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:48.318545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":17401288,"num_objects":49,"num_object_clones":17,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":20,"num_read":4618,"num_read_kb":10568,"num_write":741,"num_write_kb":5301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4454693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.537731+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:52:09.537731+0000","last_peered":"2024-09-11T05:52:09.537731+0000","last_clean":"2024-09-11T05:52:09.537731+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:52:09.537731+0000","last_undegraded":"2024-09-11T05:52:09.537731+0000","last_fullsized":"2024-09-11T05:52:09.537731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"74'335","reported_seq":2012,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524579+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:52:08.524579+0000","last_peered":"2024-09-11T05:52:08.524579+0000","last_clean":"2024-09-11T05:52:08.524579+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:52:08.524579+0000","last_undegraded":"2024-09-11T05:52:08.524579+0000","last_fullsized":"2024-09-11T05:52:08.524579+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:01.900974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":10308920,"num_objects":33,"num_object_clones":10,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":14,"num_read":2525,"num_read_kb":7930,"num_write":516,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557497+0000","last_change":"2024-09-11T05:50:58.467242+0000","last_active":"2024-09-11T05:52:09.557497+0000","last_peered":"2024-09-11T05:52:09.557497+0000","last_clean":"2024-09-11T05:52:09.557497+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:52:09.557497+0000","last_undegraded":"2024-09-11T05:52:09.557497+0000","last_fullsized":"2024-09-11T05:52:09.557497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:59.009440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"77'215","reported_seq":4712,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.683008+0000","last_change":"2024-09-11T05:52:08.530178+0000","last_active":"2024-09-11T05:52:09.683008+0000","last_peered":"2024-09-11T05:52:09.683008+0000","last_clean":"2024-09-11T05:52:09.683008+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:52:09.683008+0000","last_undegraded":"2024-09-11T05:52:09.683008+0000","last_fullsized":"2024-09-11T05:52:09.683008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:27:20.959367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":19792889,"num_objects":23,"num_object_clones":10,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":8,"num_read":4217,"num_read_kb":22070,"num_write":230,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.538008+0000","last_change":"2024-09-11T05:52:05.512835+0000","last_active":"2024-09-11T05:52:09.538008+0000","last_peered":"2024-09-11T05:52:09.538008+0000","last_clean":"2024-09-11T05:52:09.538008+0000","last_became_active":"2024-09-11T05:52:05.512365+0000","last_became_peered":"2024-09-11T05:52:05.512365+0000","last_unstale":"2024-09-11T05:52:09.538008+0000","last_undegraded":"2024-09-11T05:52:09.538008+0000","last_fullsized":"2024-09-11T05:52:09.538008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:58.934662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'292","reported_seq":3989,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.717908+0000","last_change":"2024-09-11T05:51:07.519469+0000","last_active":"2024-09-11T05:52:09.717908+0000","last_peered":"2024-09-11T05:52:09.717908+0000","last_clean":"2024-09-11T05:52:09.717908+0000","last_became_active":"2024-09-11T05:51:05.518597+0000","last_became_peered":"2024-09-11T05:51:05.518597+0000","last_unstale":"2024-09-11T05:52:09.717908+0000","last_undegraded":"2024-09-11T05:52:09.717908+0000","last_fullsized":"2024-09-11T05:52:09.717908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:54:17.463045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":50243907,"num_objects":42,"num_object_clones":15,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":16,"num_read":4870,"num_read_kb":17910,"num_write":328,"num_write_kb":8279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"77'445","reported_seq":2585,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.873362+0000","last_change":"2024-09-11T05:51:07.504814+0000","last_active":"2024-09-11T05:52:09.873362+0000","last_peered":"2024-09-11T05:52:09.873362+0000","last_clean":"2024-09-11T05:52:09.873362+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:52:09.873362+0000","last_undegraded":"2024-09-11T05:52:09.873362+0000","last_fullsized":"2024-09-11T05:52:09.873362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:32.356132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":11131449,"num_objects":41,"num_object_clones":13,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":15,"num_read":3768,"num_read_kb":8237,"num_write":682,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":131,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.567937+0000","last_change":"2024-09-11T05:50:58.174957+0000","last_active":"2024-09-11T05:52:09.567937+0000","last_peered":"2024-09-11T05:52:09.567937+0000","last_clean":"2024-09-11T05:52:09.567937+0000","last_became_active":"2024-09-11T05:50:56.159488+0000","last_became_peered":"2024-09-11T05:50:56.159488+0000","last_unstale":"2024-09-11T05:52:09.567937+0000","last_undegraded":"2024-09-11T05:52:09.567937+0000","last_fullsized":"2024-09-11T05:52:09.567937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:00.530403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001935132,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"77'434","reported_seq":2475,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.813643+0000","last_change":"2024-09-11T05:52:08.529630+0000","last_active":"2024-09-11T05:52:09.813643+0000","last_peered":"2024-09-11T05:52:09.813643+0000","last_clean":"2024-09-11T05:52:09.813643+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:52:09.813643+0000","last_undegraded":"2024-09-11T05:52:09.813643+0000","last_fullsized":"2024-09-11T05:52:09.813643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:17:39.973533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":18211408,"num_objects":46,"num_object_clones":14,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":17,"num_read":3609,"num_read_kb":9208,"num_write":526,"num_write_kb":4821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.634343+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:52:09.634343+0000","last_peered":"2024-09-11T05:52:09.634343+0000","last_clean":"2024-09-11T05:52:09.634343+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:52:09.634343+0000","last_undegraded":"2024-09-11T05:52:09.634343+0000","last_fullsized":"2024-09-11T05:52:09.634343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"77'475","reported_seq":4565,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.737806+0000","last_change":"2024-09-11T05:51:07.517865+0000","last_active":"2024-09-11T05:52:09.737806+0000","last_peered":"2024-09-11T05:52:09.737806+0000","last_clean":"2024-09-11T05:52:09.737806+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:52:09.737806+0000","last_undegraded":"2024-09-11T05:52:09.737806+0000","last_fullsized":"2024-09-11T05:52:09.737806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:45:29.452238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":28942436,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":22,"num_read":7114,"num_read_kb":17045,"num_write":601,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.556838+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:52:09.556838+0000","last_peered":"2024-09-11T05:52:09.556838+0000","last_clean":"2024-09-11T05:52:09.556838+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:52:09.556838+0000","last_undegraded":"2024-09-11T05:52:09.556838+0000","last_fullsized":"2024-09-11T05:52:09.556838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"75'321","reported_seq":6052,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.524902+0000","last_change":"2024-09-11T05:51:07.506873+0000","last_active":"2024-09-11T05:52:08.524902+0000","last_peered":"2024-09-11T05:52:08.524902+0000","last_clean":"2024-09-11T05:52:08.524902+0000","last_became_active":"2024-09-11T05:51:05.518480+0000","last_became_peered":"2024-09-11T05:51:05.518480+0000","last_unstale":"2024-09-11T05:52:08.524902+0000","last_undegraded":"2024-09-11T05:52:08.524902+0000","last_fullsized":"2024-09-11T05:52:08.524902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:32:06.752512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":38781364,"num_objects":47,"num_object_clones":17,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":16,"num_read":6194,"num_read_kb":16690,"num_write":458,"num_write_kb":11923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":130,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.567937+0000","last_change":"2024-09-11T05:50:58.173989+0000","last_active":"2024-09-11T05:52:09.567937+0000","last_peered":"2024-09-11T05:52:09.567937+0000","last_clean":"2024-09-11T05:52:09.567937+0000","last_became_active":"2024-09-11T05:50:56.157990+0000","last_became_peered":"2024-09-11T05:50:56.157990+0000","last_unstale":"2024-09-11T05:52:09.567937+0000","last_undegraded":"2024-09-11T05:52:09.567937+0000","last_fullsized":"2024-09-11T05:52:09.567937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:49.716513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095733600000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":279,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.556926+0000","last_change":"2024-09-11T05:50:52.122077+0000","last_active":"2024-09-11T05:52:09.556926+0000","last_peered":"2024-09-11T05:52:09.556926+0000","last_clean":"2024-09-11T05:52:09.556926+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:52:09.556926+0000","last_undegraded":"2024-09-11T05:52:09.556926+0000","last_fullsized":"2024-09-11T05:52:09.556926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:23.794972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"76'256","reported_seq":5071,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.534192+0000","last_change":"2024-09-11T05:52:08.528671+0000","last_active":"2024-09-11T05:52:08.534192+0000","last_peered":"2024-09-11T05:52:08.534192+0000","last_clean":"2024-09-11T05:52:08.534192+0000","last_became_active":"2024-09-11T05:52:08.527718+0000","last_became_peered":"2024-09-11T05:52:08.527718+0000","last_unstale":"2024-09-11T05:52:08.534192+0000","last_undegraded":"2024-09-11T05:52:08.534192+0000","last_fullsized":"2024-09-11T05:52:08.534192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:04:17.998135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":31506581,"num_objects":43,"num_object_clones":16,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":4520,"num_read_kb":14610,"num_write":308,"num_write_kb":16330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"74'189","reported_seq":2711,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.548263+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:52:09.548263+0000","last_peered":"2024-09-11T05:52:09.548263+0000","last_clean":"2024-09-11T05:52:09.548263+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:52:09.548263+0000","last_undegraded":"2024-09-11T05:52:09.548263+0000","last_fullsized":"2024-09-11T05:52:09.548263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:47.441213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":8665108,"num_objects":45,"num_object_clones":15,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":19,"num_read":2465,"num_read_kb":10301,"num_write":170,"num_write_kb":2686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"77'380","reported_seq":4262,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.842090+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:52:09.842090+0000","last_peered":"2024-09-11T05:52:09.842090+0000","last_clean":"2024-09-11T05:52:09.842090+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:52:09.842090+0000","last_undegraded":"2024-09-11T05:52:09.842090+0000","last_fullsized":"2024-09-11T05:52:09.842090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:05:27.672754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32153508,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":15,"num_read":5618,"num_read_kb":13156,"num_write":469,"num_write_kb":9741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"74'238","reported_seq":2180,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-11T05:52:08.525039+0000","last_change":"2024-09-11T05:51:07.505950+0000","last_active":"2024-09-11T05:52:08.525039+0000","last_peered":"2024-09-11T05:52:08.525039+0000","last_clean":"2024-09-11T05:52:08.525039+0000","last_became_active":"2024-09-11T05:51:05.511600+0000","last_became_peered":"2024-09-11T05:51:05.511600+0000","last_unstale":"2024-09-11T05:52:08.525039+0000","last_undegraded":"2024-09-11T05:52:08.525039+0000","last_fullsized":"2024-09-11T05:52:08.525039+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:53:22.714654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":26324894,"num_objects":46,"num_object_clones":16,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":18,"num_read":2233,"num_read_kb":14444,"num_write":279,"num_write_kb":7816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"77'272","reported_seq":4702,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.841547+0000","last_change":"2024-09-11T05:51:07.518494+0000","last_active":"2024-09-11T05:52:09.841547+0000","last_peered":"2024-09-11T05:52:09.841547+0000","last_clean":"2024-09-11T05:52:09.841547+0000","last_became_active":"2024-09-11T05:51:05.513377+0000","last_became_peered":"2024-09-11T05:51:05.513377+0000","last_unstale":"2024-09-11T05:52:09.841547+0000","last_undegraded":"2024-09-11T05:52:09.841547+0000","last_fullsized":"2024-09-11T05:52:09.841547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:21.993332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":10083137,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":13,"num_read":4713,"num_read_kb":7847,"num_write":366,"num_write_kb":7093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"74'238","reported_seq":3907,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557321+0000","last_change":"2024-09-11T05:51:07.538123+0000","last_active":"2024-09-11T05:52:09.557321+0000","last_peered":"2024-09-11T05:52:09.557321+0000","last_clean":"2024-09-11T05:52:09.557321+0000","last_became_active":"2024-09-11T05:51:05.528135+0000","last_became_peered":"2024-09-11T05:51:05.528135+0000","last_unstale":"2024-09-11T05:52:09.557321+0000","last_undegraded":"2024-09-11T05:52:09.557321+0000","last_fullsized":"2024-09-11T05:52:09.557321+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:02:20.555045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":15909655,"num_objects":41,"num_object_clones":14,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":16,"num_read":3952,"num_read_kb":15169,"num_write":287,"num_write_kb":4403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"74'717","reported_seq":7769,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.781078+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:52:09.781078+0000","last_peered":"2024-09-11T05:52:09.781078+0000","last_clean":"2024-09-11T05:52:09.781078+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:52:09.781078+0000","last_undegraded":"2024-09-11T05:52:09.781078+0000","last_fullsized":"2024-09-11T05:52:09.781078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:00:50.860848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":44084798,"num_objects":44,"num_object_clones":13,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":13,"num_read":9776,"num_read_kb":27990,"num_write":1023,"num_write_kb":14173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"74'351","reported_seq":2367,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.773585+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:52:09.773585+0000","last_peered":"2024-09-11T05:52:09.773585+0000","last_clean":"2024-09-11T05:52:09.773585+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:52:09.773585+0000","last_undegraded":"2024-09-11T05:52:09.773585+0000","last_fullsized":"2024-09-11T05:52:09.773585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:46:47.228514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":42069883,"num_objects":43,"num_object_clones":14,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":18,"num_read":2931,"num_read_kb":13109,"num_write":546,"num_write_kb":8021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"74'664","reported_seq":2336,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.794895+0000","last_change":"2024-09-11T05:51:12.368656+0000","last_active":"2024-09-11T05:52:09.794895+0000","last_peered":"2024-09-11T05:52:09.794895+0000","last_clean":"2024-09-11T05:52:09.794895+0000","last_became_active":"2024-09-11T05:51:12.368314+0000","last_became_peered":"2024-09-11T05:51:12.368314+0000","last_unstale":"2024-09-11T05:52:09.794895+0000","last_undegraded":"2024-09-11T05:52:09.794895+0000","last_fullsized":"2024-09-11T05:52:09.794895+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":664,"log_dups_size":0,"ondisk_log_size":664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:10.161691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861883,"num_objects":47,"num_object_clones":18,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":18,"num_read":4363,"num_read_kb":20603,"num_write":1077,"num_write_kb":9932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"77'262","reported_seq":3680,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.886405+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:52:09.886405+0000","last_peered":"2024-09-11T05:52:09.886405+0000","last_clean":"2024-09-11T05:52:09.886405+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:52:09.886405+0000","last_undegraded":"2024-09-11T05:52:09.886405+0000","last_fullsized":"2024-09-11T05:52:09.886405+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:05:30.808687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":21867925,"num_objects":54,"num_object_clones":21,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":19,"num_read":4012,"num_read_kb":10375,"num_write":316,"num_write_kb":4741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"77'487","reported_seq":2547,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.859796+0000","last_change":"2024-09-11T05:51:07.518755+0000","last_active":"2024-09-11T05:52:09.859796+0000","last_peered":"2024-09-11T05:52:09.859796+0000","last_clean":"2024-09-11T05:52:09.859796+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:52:09.859796+0000","last_undegraded":"2024-09-11T05:52:09.859796+0000","last_fullsized":"2024-09-11T05:52:09.859796+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:55:37.678628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":9355868,"num_objects":38,"num_object_clones":13,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":15,"num_read":3826,"num_read_kb":12034,"num_write":694,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"74'249","reported_seq":6989,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-11T05:52:09.557045+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:52:09.557045+0000","last_peered":"2024-09-11T05:52:09.557045+0000","last_clean":"2024-09-11T05:52:09.557045+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:52:09.557045+0000","last_undegraded":"2024-09-11T05:52:09.557045+0000","last_fullsized":"2024-09-11T05:52:09.557045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:22:32.394874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":43069232,"num_objects":50,"num_object_clones":18,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":16,"num_read":7136,"num_read_kb":16021,"num_write":338,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":783509561,"num_objects":1397,"num_object_clones":476,"num_object_copies":2794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":520,"num_read":153004,"num_read_kb":445528,"num_write":14866,"num_write_kb":239974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":26623623,"num_keys_recovered":60,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":224972800,"data_stored":295144875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11029,"ondisk_log_size":11029,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64124,"kb_used_data":37092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94307716,"statfs":{"total":96636764160,"available":96571101184,"internally_reserved":0,"allocated":37982208,"data_stored":49282787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50904,"kb_used_data":15556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320936,"statfs":{"total":96636764160,"available":96584638464,"internally_reserved":0,"allocated":15929344,"data_stored":18054539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64516,"kb_used_data":25008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94307324,"statfs":{"total":96636764160,"available":96570699776,"internally_reserved":0,"allocated":25608192,"data_stored":35230570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97872,"kb_used_data":52796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94273968,"statfs":{"total":96636764160,"available":96536543232,"internally_reserved":0,"allocated":54063104,"data_stored":73124282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70488,"kb_used_data":30980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94301352,"statfs":{"total":96636764160,"available":96564584448,"internally_reserved":0,"allocated":31723520,"data_stored":39186001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39008,"kb_used_data":7812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332832,"statfs":{"total":96636764160,"available":96596819968,"internally_reserved":0,"allocated":7999488,"data_stored":9329092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56680,"kb_used_data":21324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315160,"statfs":{"total":96636764160,"available":96578723840,"internally_reserved":0,"allocated":21835776,"data_stored":27380030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82192,"kb_used_data":37100,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94289648,"statfs":{"total":96636764160,"available":96552599552,"internally_reserved":0,"allocated":37990400,"data_stored":47848652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36532224,"data_stored":46869388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20955136,"data_stored":26993974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7110656,"data_stored":8926637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":38783546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53215232,"data_stored":72743702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24735744,"data_stored":34844496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15052800,"data_stored":17660391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36536320,"data_stored":48322741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:52:12.423 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T05:52:12.423 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:12.424 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:12.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-09-11T05:52:12.456 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.456 INFO:teuthology.orchestra.run.smithi165.stdout:4351 write 0x16573ac thru 0x1665125 (0xdd7a bytes) 2024-09-11T05:52:12.457 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.457 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.459 INFO:teuthology.orchestra.run.smithi165.stdout:4352 read 0x6df4e6f thru 0x6df9749 (0x48db bytes) 2024-09-11T05:52:12.460 INFO:teuthology.orchestra.run.smithi165.stdout:4353 read 0x66cf5cf thru 0x66d3b89 (0x45bb bytes) 2024-09-11T05:52:12.461 INFO:teuthology.orchestra.run.smithi165.stdout:4354 write 0x8704e8b thru 0x8712e35 (0xdfab bytes) 2024-09-11T05:52:12.465 INFO:teuthology.orchestra.run.smithi165.stdout:4355 read 0x9fe5c02 thru 0x9fe74fc (0x18fb bytes) 2024-09-11T05:52:12.465 INFO:teuthology.orchestra.run.smithi165.stdout:4356 write 0x97f6bd8 thru 0x97f78ff (0xd28 bytes) 2024-09-11T05:52:12.467 INFO:teuthology.orchestra.run.smithi165.stdout:4360 read 0x190cd62 thru 0x191949f (0xc73e bytes) 2024-09-11T05:52:12.468 INFO:teuthology.orchestra.run.smithi165.stdout:4361 read 0x1dc43ae thru 0x1dceb8d (0xa7e0 bytes) 2024-09-11T05:52:12.469 INFO:teuthology.orchestra.run.smithi165.stdout:4362 writesame 0x9eb6688 thru 0x9ec3f6a data_size 0x1(0xd8e3 bytes) 2024-09-11T05:52:12.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:12.494 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:12.492+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:12.529 INFO:teuthology.orchestra.run.smithi165.stdout:4365 read 0xbc39570 thru 0xbc3a76b (0x11fc bytes) 2024-09-11T05:52:12.554 INFO:teuthology.orchestra.run.smithi165.stdout:4366 trunc from 0xc2e08b7 to 0x12ea4e5 2024-09-11T05:52:12.574 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.595 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:12.592+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:12.605 INFO:teuthology.orchestra.run.smithi165.stdout:4368 writesame 0xac093f6 thru 0xac12ebf data_size 0x1(0x9aca bytes) 2024-09-11T05:52:12.616 INFO:teuthology.orchestra.run.smithi165.stdout:4369 write 0xb6c3925 thru 0xb6c9c3c (0x6318 bytes) 2024-09-11T05:52:12.618 INFO:teuthology.orchestra.run.smithi165.stdout:4371 write 0xb6a6eae thru 0xb6b1667 (0xa7ba bytes) 2024-09-11T05:52:12.621 INFO:teuthology.orchestra.run.smithi165.stdout:4372 punch from 0x19e382e to 0x19f206b, (0xe83d bytes) 2024-09-11T05:52:12.627 INFO:teuthology.orchestra.run.smithi165.stdout:4373 trunc from 0xb6c9c3d to 0x7a77cbb 2024-09-11T05:52:12.649 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.649 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.651 INFO:teuthology.orchestra.run.smithi165.stdout:4374 write 0x8832f95 thru 0x88406e0 (0xd74c bytes) 2024-09-11T05:52:12.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:12.670 INFO:teuthology.orchestra.run.smithi165.stdout:4375 write 0xcc277ba thru 0xcc27bae (0x3f5 bytes) 2024-09-11T05:52:12.672 INFO:teuthology.orchestra.run.smithi165.stdout:4376 writesame 0x9fb6edf thru 0x9fc2f63 data_size 0x1(0xc085 bytes) 2024-09-11T05:52:12.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:12.710 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:12.707+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:12.720 INFO:teuthology.orchestra.run.smithi165.stdout:4378 punch from 0x18cbedd to 0x18d09d5, (0x4af8 bytes) 2024-09-11T05:52:12.728 INFO:teuthology.orchestra.run.smithi165.stdout:4379 trunc from 0xcc27baf to 0x7818384 2024-09-11T05:52:12.742 INFO:teuthology.orchestra.run.smithi165.stdout:4380 read 0x470b76b thru 0x470ee16 (0x36ac bytes) 2024-09-11T05:52:12.743 INFO:teuthology.orchestra.run.smithi165.stdout:4381 punch from 0x6e5cb60 to 0x6e65dda, (0x927a bytes) 2024-09-11T05:52:12.743 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.743 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.743 INFO:teuthology.orchestra.run.smithi165.stdout:4384 write 0x6dfae45 thru 0x6dfbc83 (0xe3f bytes) 2024-09-11T05:52:12.754 INFO:teuthology.orchestra.run.smithi165.stdout:4387 trunc from 0x7818384 to 0xd712129 2024-09-11T05:52:12.763 INFO:teuthology.orchestra.run.smithi165.stdout:4389 read 0xaa5e517 thru 0xaa648c8 (0x63b2 bytes) 2024-09-11T05:52:12.764 INFO:teuthology.orchestra.run.smithi165.stdout:4391 write 0x4d51ee8 thru 0x4d5ec47 (0xcd60 bytes) 2024-09-11T05:52:12.766 INFO:teuthology.orchestra.run.smithi165.stdout:4392 read 0x3350e32 thru 0x33599d9 (0x8ba8 bytes) 2024-09-11T05:52:12.767 INFO:teuthology.orchestra.run.smithi165.stdout:4393 write 0x84cf596 thru 0x84d8154 (0x8bbf bytes) 2024-09-11T05:52:12.769 INFO:teuthology.orchestra.run.smithi165.stdout:4394 read 0x3468f16 thru 0x346ebff (0x5cea bytes) 2024-09-11T05:52:12.770 INFO:teuthology.orchestra.run.smithi165.stdout:4395 write 0x84047a3 thru 0x8405dbe (0x161c bytes) 2024-09-11T05:52:12.772 INFO:teuthology.orchestra.run.smithi165.stdout:4399 read 0xca8a28b thru 0xca8e1a0 (0x3f16 bytes) 2024-09-11T05:52:12.773 INFO:teuthology.orchestra.run.smithi165.stdout:4400 punch from 0xcb58269 to 0xcb6358a, (0xb321 bytes) 2024-09-11T05:52:12.773 INFO:teuthology.orchestra.run.smithi165.stdout:4401 write 0x63d0e0d thru 0x63d9cdc (0x8ed0 bytes) 2024-09-11T05:52:12.790 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.790 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.791 INFO:teuthology.orchestra.run.smithi165.stdout:4402 trunc from 0xd712129 to 0x2d30c79 2024-09-11T05:52:12.793 INFO:tasks.ceph.mon.a.smithi165.stderr:2024-09-11T05:52:12.792+0000 7f2b6abd3640 -1 mon.a@0(leader).osd e79 definitely_dead 0 2024-09-11T05:52:12.797 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:12.796+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:12.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:12.808 INFO:teuthology.orchestra.run.smithi165.stdout:4404 punch from 0xe3feed to 0xe475f6, (0x7709 bytes) 2024-09-11T05:52:12.827 INFO:teuthology.orchestra.run.smithi165.stdout:4406 write 0xacdb655 thru 0xace39d9 (0x8385 bytes) 2024-09-11T05:52:12.832 INFO:teuthology.orchestra.run.smithi165.stdout:4408 trunc from 0xace39da to 0x97efd78 2024-09-11T05:52:12.853 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:12.853 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:12.858 INFO:teuthology.orchestra.run.smithi165.stdout:4409 write 0xcc1aefa thru 0xcc235b4 (0x86bb bytes) 2024-09-11T05:52:12.879 INFO:teuthology.orchestra.run.smithi165.stdout:4410 trunc from 0xcc235b5 to 0xf3dbe3 2024-09-11T05:52:12.884 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:12.884 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:12.891 INFO:teuthology.orchestra.run.smithi165.stdout:4412 punch from 0xb10b49 to 0xb1167e, (0xb35 bytes) 2024-09-11T05:52:12.891 INFO:teuthology.orchestra.run.smithi165.stdout:4413 read 0x94f3c3 thru 0x9502ba (0xef8 bytes) 2024-09-11T05:52:12.892 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.892 INFO:teuthology.orchestra.run.smithi165.stdout:4416 read 0xb9c11b thru 0xb9cda0 (0xc86 bytes) 2024-09-11T05:52:12.892 INFO:teuthology.orchestra.run.smithi165.stdout:4417 read 0xef18ae thru 0xefa1f8 (0x894b bytes) 2024-09-11T05:52:12.892 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:12.892 INFO:teuthology.orchestra.run.smithi165.stdout:4420 punch from 0xeaa15c to 0xeb7f30, (0xddd4 bytes) 2024-09-11T05:52:12.898 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:12.896+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:12.902 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:12.902 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:12.919 INFO:teuthology.orchestra.run.smithi165.stdout:4421 write 0xea871cb thru 0xea94502 (0xd338 bytes) 2024-09-11T05:52:12.921 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:12.921 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:12.922 INFO:teuthology.orchestra.run.smithi165.stdout:4422 write 0x2cb67a6 thru 0x2cb6c5c (0x4b7 bytes) 2024-09-11T05:52:12.923 INFO:teuthology.orchestra.run.smithi165.stdout:4423 writesame 0x53edbfc thru 0x53f9542 data_size 0x1(0xb947 bytes) 2024-09-11T05:52:12.939 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:12.940 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:12.955 INFO:teuthology.orchestra.run.smithi165.stdout:4424 read 0x51931c3 thru 0x51a0297 (0xd0d5 bytes) 2024-09-11T05:52:12.957 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:12.957 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:12.957 INFO:teuthology.orchestra.run.smithi165.stdout:4425 write 0x9a85b96 thru 0x9a94d8e (0xf1f9 bytes) 2024-09-11T05:52:12.971 INFO:teuthology.orchestra.run.smithi165.stdout:4426 trunc from 0xea94503 to 0x6fe6fb7 2024-09-11T05:52:12.977 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:12.977 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:12.986 INFO:teuthology.orchestra.run.smithi165.stdout:4427 read 0x11bae9e thru 0x11c1188 (0x62eb bytes) 2024-09-11T05:52:12.987 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:12.987 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:12.996 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:12.996 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:12.998 INFO:teuthology.orchestra.run.smithi165.stdout:4428 trunc from 0x6fe6fb7 to 0xd3f2163 2024-09-11T05:52:12.998 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:12.995+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:13.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:13.006 INFO:teuthology.orchestra.run.smithi165.stdout:4429 write 0x2602240 thru 0x260db15 (0xb8d6 bytes) 2024-09-11T05:52:13.009 INFO:teuthology.orchestra.run.smithi165.stdout:4430 writesame 0x444cc43 thru 0x4453b81 data_size 0xb(0x6f3f bytes) 2024-09-11T05:52:13.013 INFO:teuthology.orchestra.run.smithi165.stdout:4431 read 0x9a263b1 thru 0x9a2b91c (0x556c bytes) 2024-09-11T05:52:13.014 INFO:teuthology.orchestra.run.smithi165.stdout:4432 punch from 0x5cfdede to 0x5cfdf44, (0x66 bytes) 2024-09-11T05:52:13.014 INFO:teuthology.orchestra.run.smithi165.stdout:4433 read 0x9f233cc thru 0x9f31b06 (0xe73b bytes) 2024-09-11T05:52:13.033 INFO:teuthology.orchestra.run.smithi165.stdout:4434 trunc from 0xd3f2163 to 0x2ef83af 2024-09-11T05:52:13.048 INFO:teuthology.orchestra.run.smithi165.stdout:4436 read 0x99a4a5 thru 0x99f050 (0x4bac bytes) 2024-09-11T05:52:13.048 INFO:teuthology.orchestra.run.smithi165.stdout:4437 read 0x3949e7 thru 0x3a3fcd (0xf5e7 bytes) 2024-09-11T05:52:13.071 INFO:teuthology.orchestra.run.smithi165.stdout:4438 writesame 0x90d3d99 thru 0x90de2b4 data_size 0x2(0xa51c bytes) 2024-09-11T05:52:13.073 INFO:teuthology.orchestra.run.smithi165.stdout:4440 writesame 0x3104548 thru 0x31101de data_size 0x85(0xbc97 bytes) 2024-09-11T05:52:13.076 INFO:teuthology.orchestra.run.smithi165.stdout:4441 punch from 0x5becd57 to 0x5bf943e, (0xc6e7 bytes) 2024-09-11T05:52:13.076 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.076 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.076 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.076 INFO:teuthology.orchestra.run.smithi165.stdout:4446 write 0x6d23b5 thru 0x6dfc2b (0xd877 bytes) 2024-09-11T05:52:13.079 INFO:teuthology.orchestra.run.smithi165.stdout:4447 punch from 0x4ac997 to 0x4adf0a, (0x1573 bytes) 2024-09-11T05:52:13.079 INFO:teuthology.orchestra.run.smithi165.stdout:4449 writesame 0x48b24b6 thru 0x48b740f data_size 0x7(0x4f5a bytes) 2024-09-11T05:52:13.082 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.082 INFO:teuthology.orchestra.run.smithi165.stdout:4453 writesame 0x741feb3 thru 0x742967a data_size 0xc(0x97c8 bytes) 2024-09-11T05:52:13.100 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:13.100+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:13.101 INFO:teuthology.orchestra.run.smithi165.stdout:4454 write 0xa925777 thru 0xa926ee7 (0x1771 bytes) 2024-09-11T05:52:13.116 INFO:teuthology.orchestra.run.smithi165.stdout:4456 writesame 0xc43961a thru 0xc43e255 data_size 0x52(0x4c3c bytes) 2024-09-11T05:52:13.119 INFO:teuthology.orchestra.run.smithi165.stdout:4459 punch from 0x4414561 to 0x4416a79, (0x2518 bytes) 2024-09-11T05:52:13.139 INFO:teuthology.orchestra.run.smithi165.stdout:4460 trunc from 0xc43e256 to 0x189c862 2024-09-11T05:52:13.159 INFO:teuthology.orchestra.run.smithi165.stdout:4461 read 0x1513b37 thru 0x1522b4c (0xf016 bytes) 2024-09-11T05:52:13.160 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:12378 3.11 3:8fd7d621:::rbd_data.1124518b2c12.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.06840175, 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.070297296999999995, 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 12378 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105094+0000", 2024-09-11T05:52:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105096+0000", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.77e-06 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105105+0000", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5729999999999997e-06 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105131+0000", 2024-09-11T05:52:13.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5700000000000001e-05 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105145+0000", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.358e-05 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105297+0000", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015220199999999999 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.105443+0000", 2024-09-11T05:52:13.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014585499999999999 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.116910+0000", 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011467221999999999 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175180+0000", 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058270303000000002 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:13.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175217+0000", 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6956999999999999e-05 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175307+0000", 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9586000000000006e-05 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:22.175391+0000", 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549000000000007e-05 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65322 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1427456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.826814366999997, 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064399524, 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65322 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346681+0000", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346682+0000", 2024-09-11T05:52:13.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7800000000000002e-07 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346684+0000", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346691+0000", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7159999999999997e-06 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.346696+0000", 2024-09-11T05:52:13.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2750000000000003e-06 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410917+0000", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064220632 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.410990+0000", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2935000000000006e-05 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411081+0000", 2024-09-11T05:52:13.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1127999999999999e-05 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65334 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.826203540999998, 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064338716000000004, 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.186 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65334 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347292+0000", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347293+0000", 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2600000000000001e-07 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347294+0000", 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7400000000000001e-06 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347298+0000", 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2339999999999999e-06 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347301+0000", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.805e-06 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411479+0000", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064177351999999993 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411566+0000", 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7140999999999998e-05 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411631+0000", 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5217999999999998e-05 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65336 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 184320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.826179134, 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064131940999999998, 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65336 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347317+0000", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3799999999999999e-07 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347318+0000", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347321+0000", 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.162e-06 2024-09-11T05:52:13.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347324+0000", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3740000000000001e-06 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411298+0000", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063974520000000007 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411366+0000", 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7433000000000003e-05 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.411449+0000", 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3088000000000001e-05 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65337 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1505280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.826168815999999, 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064848631000000004, 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.194 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65337 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.347327+0000", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349694+0000", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023672049999999998 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349700+0000", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6470000000000003e-06 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349710+0000", 2024-09-11T05:52:13.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0416e-05 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349728+0000", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7215999999999999e-05 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412023+0000", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062295229000000001 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412098+0000", 2024-09-11T05:52:13.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5179000000000006e-05 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.412176+0000", 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7738999999999995e-05 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65352 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 466944~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823565201000001, 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063986790000000002, 2024-09-11T05:52:13.198 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65352 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349931+0000", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2399999999999999e-07 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349932+0000", 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.11e-06 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349935+0000", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1540000000000002e-06 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349938+0000", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413768+0000", 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063830260999999999 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413837+0000", 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9231000000000002e-05 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413917+0000", 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0290000000000005e-05 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.202 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65353 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 491520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823554758999997, 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064145419999999995, 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65353 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349941+0000", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2999999999999999e-07 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349942+0000", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349946+0000", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.061e-06 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349949+0000", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8149999999999999e-06 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.413950+0000", 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064000776999999995 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414018+0000", 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7935999999999995e-05 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414087+0000", 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65355 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 522240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823543051999998, 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064313043, 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65355 2024-09-11T05:52:13.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349953+0000", 2024-09-11T05:52:13.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.28e-07 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349954+0000", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349957+0000", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349960+0000", 2024-09-11T05:52:13.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7590000000000002e-06 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414117+0000", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064156737000000005 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414200+0000", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3511999999999996e-05 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414266+0000", 2024-09-11T05:52:13.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5393000000000002e-05 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65356 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 546816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823531971999998, 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064502437999999995, 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.211 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65356 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349964+0000", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349966+0000", 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7209999999999999e-06 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349974+0000", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6890000000000004e-06 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349990+0000", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6742999999999999e-05 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414297+0000", 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064306181000000004 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414400+0000", 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103732 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414466+0000", 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5925000000000003e-05 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65357 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 571392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.215 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823496876999997, 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064644852000000003, 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65357 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.349999+0000", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350000+0000", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1200000000000003e-07 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350002+0000", 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6879999999999999e-06 2024-09-11T05:52:13.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350016+0000", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3764999999999999e-05 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350022+0000", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.49e-06 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414509+0000", 2024-09-11T05:52:13.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064487078000000003 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414580+0000", 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1625999999999995e-05 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414644+0000", 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3592999999999999e-05 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65359 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 595968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823456370999999, 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064817939000000005, 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65359 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350039+0000", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350040+0000", 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2099999999999996e-07 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350043+0000", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9679999999999998e-06 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350052+0000", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350065+0000", 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3023999999999999e-05 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414701+0000", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064636009999999994 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414775+0000", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3888999999999997e-05 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414857+0000", 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2817000000000003e-05 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65361 3.d 3:b1b026c0:::rbd_data.1124b2f65e38.0000000000000016:11 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823422575000002, 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065560283999999996, 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65361 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350073+0000", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350074+0000", 2024-09-11T05:52:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0699999999999997e-07 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350076+0000", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4059999999999999e-06 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350084+0000", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350104+0000", 2024-09-11T05:52:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695999999999999e-05 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415495+0000", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065391447000000005 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415562+0000", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7275999999999998e-05 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415634+0000", 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1082999999999997e-05 2024-09-11T05:52:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65366 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823383671999999, 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064926965000000003, 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65366 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350112+0000", 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350113+0000", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.44e-07 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350127+0000", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4439e-05 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350134+0000", 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1590000000000004e-06 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350140+0000", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3940000000000004e-06 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414884+0000", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064744064000000004 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.414970+0000", 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6693000000000005e-05 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415039+0000", 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8671999999999996e-05 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65380 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1677312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823297171, 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065060571999999997, 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65380 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350199+0000", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5999999999999999e-07 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350201+0000", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1339999999999998e-06 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350207+0000", 2024-09-11T05:52:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3279999999999996e-06 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350210+0000", 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649999999999999e-06 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415095+0000", 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064884598000000002 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415192+0000", 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7042999999999999e-05 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415259+0000", 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7544000000000002e-05 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65395 3.15 3:ae3e5bb0:::rbd_data.1124a9e466a.0000000000000007:10 [read 1755136~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823279909999997, 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221741999999999, 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65395 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350216+0000", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.84e-07 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350218+0000", 2024-09-11T05:52:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9e-06 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350235+0000", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6949e-05 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350238+0000", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2160000000000002e-06 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415287+0000", 2024-09-11T05:52:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065048724000000002 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415357+0000", 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9917000000000002e-05 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415438+0000", 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0652e-05 2024-09-11T05:52:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65448 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.822973836000003, 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172466999999998, 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65448 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350522+0000", 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350523+0000", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.75e-07 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350531+0000", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350539+0000", 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2090000000000005e-06 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350545+0000", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9370000000000001e-06 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415686+0000", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065140919000000005 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415694+0000", 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9080000000000001e-06 2024-09-11T05:52:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65451 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.822914062999999, 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065172636000000006, 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65451 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350582+0000", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.96e-07 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350584+0000", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1210000000000001e-06 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350590+0000", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0549999999999996e-06 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350596+0000", 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6280000000000002e-06 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415740+0000", 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065144118000000001 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415754+0000", 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4318000000000001e-05 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65452 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.822892420000002, 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065203536000000006, 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65452 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350603+0000", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350604+0000", 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350606+0000", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0710000000000002e-06 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350612+0000", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350617+0000", 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469999999999998e-06 2024-09-11T05:52:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415798+0000", 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065181801999999997 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415807+0000", 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4740000000000005e-06 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.252 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65454 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.822871216999999, 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065221873999999999, 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65454 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350625+0000", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350627+0000", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.03e-06 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.350633+0000", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8950000000000003e-06 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351646+0000", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010134759999999999 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415839+0000", 2024-09-11T05:52:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064192704000000003 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415847+0000", 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4080000000000001e-06 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:65455 3.d 3.3640d8d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.821826426000001, 2024-09-11T05:52:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064208996000000004, 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65455 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351669+0000", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351670+0000", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-07 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351675+0000", 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.296e-06 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351685+0000", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0077e-05 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.351693+0000", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0970000000000006e-06 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415871+0000", 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064178624000000004 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.415878+0000", 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0899999999999999e-06 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:13.260 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:13.261 INFO:teuthology.orchestra.run.smithi165.stdout:4464 write 0x6e949e2 thru 0x6e993bc (0x49db bytes) 2024-09-11T05:52:13.261 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.261 INFO:teuthology.orchestra.run.smithi165.stdout:4466 write 0xa6b619a thru 0xa6bc0a5 (0x5f0c bytes) 2024-09-11T05:52:13.262 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.262 INFO:teuthology.orchestra.run.smithi165.stdout:4470 trunc from 0xa6bc0a6 to 0xee166 2024-09-11T05:52:13.262 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:13.260+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:13.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:13.293 INFO:teuthology.orchestra.run.smithi165.stdout:4471 punch from 0xb48bc to 0xbbb14, (0x7258 bytes) 2024-09-11T05:52:13.312 INFO:teuthology.orchestra.run.smithi165.stdout:4473 write 0x37b944f thru 0x37b9956 (0x508 bytes) 2024-09-11T05:52:13.353 INFO:teuthology.orchestra.run.smithi165.stdout:4474 writesame 0xc433e9b thru 0xc443533 data_size 0xb(0xf699 bytes) 2024-09-11T05:52:13.353 INFO:teuthology.orchestra.run.smithi165.stdout:4475 read 0x23e9d19 thru 0x23f3fd7 (0xa2bf bytes) 2024-09-11T05:52:13.353 INFO:teuthology.orchestra.run.smithi165.stdout:4477 read 0x21ad51a thru 0x21b025b (0x2d42 bytes) 2024-09-11T05:52:13.353 INFO:teuthology.orchestra.run.smithi165.stdout:4478 write 0x9c49b08 thru 0x9c4afbb (0x14b4 bytes) 2024-09-11T05:52:13.360 INFO:teuthology.orchestra.run.smithi165.stdout:4479 writesame 0x84b7169 thru 0x84c3f63 data_size 0xbd(0xcdfb bytes) 2024-09-11T05:52:13.362 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:13.359+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:13.368 INFO:teuthology.orchestra.run.smithi165.stdout:4480 read 0x27b0e33 thru 0x27bc937 (0xbb05 bytes) 2024-09-11T05:52:13.368 INFO:teuthology.orchestra.run.smithi165.stdout:4481 writesame 0x9d8c9b6 thru 0x9d99d7d data_size 0x36(0xd3c8 bytes) 2024-09-11T05:52:13.373 INFO:teuthology.orchestra.run.smithi165.stdout:4482 write 0x671c003 thru 0x6723fa9 (0x7fa7 bytes) 2024-09-11T05:52:13.395 INFO:teuthology.orchestra.run.smithi165.stdout:4484 trunc from 0xc443534 to 0x653c391 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:124905 3.1b 3:ddaaf266:::rbd_object_map.11243e6dddb9:head [call lock.assert_locked in=44b,call rbd.object_map_resize in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:13.405124+0000", 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00056042699999999995, 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060672600000000003, 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.405 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 124905 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405124+0000", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405124+0000", 2024-09-11T05:52:13.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405127+0000", 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1569999999999998e-06 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405138+0000", 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405167+0000", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8606e-05 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405181+0000", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3876000000000001e-05 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405257+0000", 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6038999999999995e-05 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405352+0000", 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5493000000000002e-05 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.405579+0000", 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022627500000000001 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-11T05:52:13.409 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:13.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:13.430 INFO:teuthology.orchestra.run.smithi165.stdout:4485 writesame 0x7152b09 thru 0x715e187 data_size 0x57(0xb67f bytes) 2024-09-11T05:52:13.456 INFO:teuthology.orchestra.run.smithi165.stdout:4487 write 0xcde4c1d thru 0xcdec1e6 (0x75ca bytes) 2024-09-11T05:52:13.460 INFO:teuthology.orchestra.run.smithi165.stdout:4489 write 0xc1d8174 thru 0xc1dc8e5 (0x4772 bytes) 2024-09-11T05:52:13.462 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:13.459+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:13.463 INFO:teuthology.orchestra.run.smithi165.stdout:4490 punch from 0x282ec8f to 0x283292a, (0x3c9b bytes) 2024-09-11T05:52:13.464 INFO:teuthology.orchestra.run.smithi165.stdout:4491 read 0x37b40b5 thru 0x37b8c50 (0x4b9c bytes) 2024-09-11T05:52:13.464 INFO:teuthology.orchestra.run.smithi165.stdout:4493 write 0x945d388 thru 0x945ed25 (0x199e bytes) 2024-09-11T05:52:13.467 INFO:teuthology.orchestra.run.smithi165.stdout:4494 punch from 0x8b3b248 to 0x8b47fe3, (0xcd9b bytes) 2024-09-11T05:52:13.467 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.467 INFO:teuthology.orchestra.run.smithi165.stdout:4497 write 0x7e7cae4 thru 0x7e7ebf3 (0x2110 bytes) 2024-09-11T05:52:13.471 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.482 INFO:teuthology.orchestra.run.smithi165.stdout:4501 trunc from 0xcdec1e7 to 0x6b6bf8d 2024-09-11T05:52:13.501 INFO:teuthology.orchestra.run.smithi165.stdout:4502 write 0x5c54f81 thru 0x5c5f524 (0xa5a4 bytes) 2024-09-11T05:52:13.532 INFO:teuthology.orchestra.run.smithi165.stdout:4503 writesame 0xd76fa60 thru 0xd779ec5 data_size 0xb(0xa466 bytes) 2024-09-11T05:52:13.534 INFO:teuthology.orchestra.run.smithi165.stdout:4504 punch from 0x4546b7d to 0x4552526, (0xb9a9 bytes) 2024-09-11T05:52:13.534 INFO:teuthology.orchestra.run.smithi165.stdout:4506 read 0x992b50a thru 0x9936dc6 (0xb8bd bytes) 2024-09-11T05:52:13.535 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.535 INFO:teuthology.orchestra.run.smithi165.stdout:4511 read 0x7075756 thru 0x70805a6 (0xae51 bytes) 2024-09-11T05:52:13.536 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.536 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:13.536 INFO:teuthology.orchestra.run.smithi165.stdout:4514 write 0x4d3144 thru 0x4e032f (0xd1ec bytes) 2024-09-11T05:52:13.544 INFO:teuthology.orchestra.run.smithi165.stdout:4516 trunc from 0xd779ec6 to 0xe93e509 2024-09-11T05:52:13.558 INFO:teuthology.orchestra.run.smithi165.stdout:4517 punch from 0x207d525 to 0x208bbc7, (0xe6a2 bytes) 2024-09-11T05:52:13.558 INFO:teuthology.orchestra.run.smithi165.stdout:4518 read 0x830c854 thru 0x830f4b2 (0x2c5f bytes) 2024-09-11T05:52:13.558 INFO:teuthology.orchestra.run.smithi165.stdout:4520 punch from 0xbf3ce80 to 0xbf48f6e, (0xc0ee bytes) 2024-09-11T05:52:13.563 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:13.560+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:13.571 INFO:teuthology.orchestra.run.smithi165.stdout:4521 trunc from 0xe93e509 to 0x708bf06 2024-09-11T05:52:13.585 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:13.585 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:13.585 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:13.585 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:13.585 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:13.587 INFO:teuthology.orchestra.run.smithi165.stderr:marked out osd.5. 2024-09-11T05:52:13.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:52:13.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:13.665 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:13.664+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:13.765 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:13.759+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:13.783 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978174985999999, 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:13.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978148007000001, 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978117747999999, 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.792 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978072463, 2024-09-11T05:52:13.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:13.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978039596999999, 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.801 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:13.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:13.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:13.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.978011542000001, 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.809 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977984476, 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:13.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:13.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.813 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977934475000001, 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.814 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:13.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:13.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:13.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977901992, 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.818 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:13.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:13.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:13.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977873523, 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.822 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977844644000001, 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:13.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977787826, 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.830 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:13.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:13.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:13.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977756794000001, 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:13.834 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.838 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977728774999999, 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:13.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977701845999999, 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:13.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:13.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:13.846 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.97767528, 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.847 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977638069000001, 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.851 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:13.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:13.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977609516000001, 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:13.855 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977561054999999, 2024-09-11T05:52:13.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:13.863 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.977522440000001, 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:13.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:13.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:13.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:13.867 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:13.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:13.869 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:13.868+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:13.970 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:13.968+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:14.035 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.036 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:14.036 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:14.036 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:14.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:14.092 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:14.092+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:14.171 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:14.168+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:14.217 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:14.217 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:14.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:14.273 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:14.272+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:14.373 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:14.372+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:14.397 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047531925999998, 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047502516999998, 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047482459999998, 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047441073000002, 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047408591, 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047388535000003, 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047369033000003, 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047344224, 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047325075000003, 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047304261000001, 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047284427999998, 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047264488000003, 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047238366999999, 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:14.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.047201782999998, 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:14.453 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:14.457 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.042413984, 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:14.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:14.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.461 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.042339237, 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:14.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:14.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:14.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.042266478000002, 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.042245557999998, 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.473 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.700279367, 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:14.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:14.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:14.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:14.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:14.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.9918399559999997, 2024-09-11T05:52:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:14.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:14.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.483 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.484 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.484 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:14.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:14.485 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:14.483+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:14.586 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:14.584+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:14.602 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:52:14.662 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.662 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:14.663 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:14.663 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:14.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:14.724 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:14.724+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:14.788 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:14.783+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:14.846 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:14.847 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:14.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:14.889 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:14.883+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.96047179, 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:14.988 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.469148592000003, 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:14.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:14.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:14.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.606171818, 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:14.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:14.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:14.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84925 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 114688~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.433617725000001, 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005558884, 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:14.999 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84925 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4000000000000002e-07 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547264+0000", 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547271+0000", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547276+0000", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7520000000000002e-06 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552413+0000", 2024-09-11T05:52:15.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051366579999999997 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552501+0000", 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8220000000000006e-05 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552820+0000", 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031835400000000002 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.43359491, 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:15.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:15.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:15.007 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.433443391000001, 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.008 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.433400233, 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:15.012 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:15.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.433374408999999, 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.017 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:15.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:15.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.325901139999999, 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.021 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:15.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90109 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.194250385, 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055123259999999997, 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90109 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786632+0000", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6969999999999999e-06 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786650+0000", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8578999999999999e-05 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786697+0000", 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-11T05:52:15.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786717+0000", 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9579999999999999e-05 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786863+0000", 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014619399999999999 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787123+0000", 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026025400000000002 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787432+0000", 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030917899999999998 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.790614+0000", 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031820160000000002 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.791881+0000", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001266585 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792030+0000", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014938199999999999 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792140+0000", 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110152 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90695 3.0 3:03391bbf:::rbd_data.11242d2cfbf4.0000000000000079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.516620061000001, 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059067149999999999, 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90695 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464262+0000", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464277+0000", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-05 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464325+0000", 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8881000000000002e-05 2024-09-11T05:52:15.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464351+0000", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5269000000000002e-05 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464527+0000", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017588099999999999 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464774+0000", 2024-09-11T05:52:15.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024752500000000001 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.465110+0000", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033546800000000001 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.467391+0000", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002281047 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470017+0000", 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026263079999999999 2024-09-11T05:52:15.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470097+0000", 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0584e-05 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470165+0000", 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7655000000000002e-05 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.516477515999998, 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:15.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.170142021, 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:15.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:15.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:15.050 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.152153874, 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:15.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.090054588999999, 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:15.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:15.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:15.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.089942092999999, 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:15.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.609997805000001, 2024-09-11T05:52:15.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:15.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:15.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:15.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:15.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.608423418999999, 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:15.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:15.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.074 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.075614376000001, 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:15.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.585326701, 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:15.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:15.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:15.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:15.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:15.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.087 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:15.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:15.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:15.089 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:15.088+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:15.190 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:15.187+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:124989 3.5 3.d0e74905 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e80)", 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:13.591720+0000", 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.664892993, 2024-09-11T05:52:15.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649186660000002, 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for peered", 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 124989 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591720+0000", 2024-09-11T05:52:15.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591720+0000", 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591723+0000", 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1530000000000001e-06 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591743+0000", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0426000000000001e-05 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591804+0000", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1038999999999996e-05 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591835+0000", 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0269000000000001e-05 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591938+0000", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010338199999999999 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591946+0000", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6599999999999995e-06 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.594380+0000", 2024-09-11T05:52:15.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024339940000000001 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.594395+0000", 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5849999999999999e-05 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600735+0000", 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.006340045 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600752+0000", 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6531000000000001e-05 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:124990 3.5 3.d0e74905 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:13.591894+0000", 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.6647181639999999, 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6648300150000002, 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for peered", 2024-09-11T05:52:15.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 124990 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591894+0000", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591894+0000", 2024-09-11T05:52:15.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591896+0000", 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6700000000000001e-06 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591904+0000", 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2840000000000006e-06 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591930+0000", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5972000000000001e-05 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.591947+0000", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6529999999999999e-05 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.592008+0000", 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1594999999999996e-05 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.592013+0000", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3340000000000003e-06 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.594424+0000", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024110709999999999 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:13.594428+0000", 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5480000000000003e-06 2024-09-11T05:52:15.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600787+0000", 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.00635852 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600798+0000", 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1428e-05 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:124991 3.5 3.d0e74905 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:14.349203+0000", 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.90740982400000003, 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90757166499999997, 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for peered", 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 124991 2024-09-11T05:52:15.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349203+0000", 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349203+0000", 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349206+0000", 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5470000000000001e-06 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349220+0000", 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4136e-05 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349256+0000", 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5593000000000003e-05 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349274+0000", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8318999999999999e-05 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349389+0000", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114298 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.349401+0000", 2024-09-11T05:52:15.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2086000000000001e-05 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600832+0000", 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25143099200000002 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600842+0000", 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0635999999999999e-05 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:124992 3.5 3.d0e74905 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:14.596889+0000", 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.65972398200000004, 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.65992738500000003, 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for peered", 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 124992 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.273 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596889+0000", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596889+0000", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596891+0000", 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4619999999999999e-06 2024-09-11T05:52:15.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596899+0000", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6199999999999999e-06 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596919+0000", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9899999999999999e-05 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.596929+0000", 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0577999999999999e-05 2024-09-11T05:52:15.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.597048+0000", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118712 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.597062+0000", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3777e-05 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600876+0000", 2024-09-11T05:52:15.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038138650000000001 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:14.600887+0000", 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0984e-05 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 4 2024-09-11T05:52:15.277 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:15.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:15.280 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:15.280 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:15.297 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:15.297 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:15.297 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:15.296+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:15.315 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:15.315 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:15.332 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:15.332 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:15.352 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:15.353 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:15.373 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:15.374 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:15.392 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:15.387+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:15.394 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:15.394 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:15.397 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:15.397 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:15.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:15.415 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:15.415 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:15.493 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:15.492+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:15.536 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:15.536 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:15.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:15.598 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:15.595+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:15.692 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:15.692 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:15.694 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:15.691+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:15.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:15.754 INFO:teuthology.orchestra.run.smithi165.stdout:4522 write 0x50d9b9f thru 0x50e54dd (0xb93f bytes) 2024-09-11T05:52:15.758 INFO:teuthology.orchestra.run.smithi165.stdout:4523 read 0x30b027b thru 0x30b53de (0x5164 bytes) 2024-09-11T05:52:15.758 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:15.758 INFO:teuthology.orchestra.run.smithi165.stdout:4525 read 0x20d0d1d thru 0x20df816 (0xeafa bytes) 2024-09-11T05:52:15.758 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:15.768 INFO:teuthology.orchestra.run.smithi165.stdout:4527 writesame 0x73482d2 thru 0x7356883 data_size 0x1(0xe5b2 bytes) 2024-09-11T05:52:15.769 INFO:teuthology.orchestra.run.smithi165.stdout:4528 read 0x62fb4f5 thru 0x63041d4 (0x8ce0 bytes) 2024-09-11T05:52:15.769 INFO:teuthology.orchestra.run.smithi165.stdout:4530 write 0x1b1b2e8 thru 0x1b20396 (0x50af bytes) 2024-09-11T05:52:15.788 INFO:teuthology.orchestra.run.smithi165.stdout:4531 write 0xa89113a thru 0xa89bcff (0xabc6 bytes) 2024-09-11T05:52:15.794 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:15.792+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:15.797 INFO:teuthology.orchestra.run.smithi165.stdout:4532 punch from 0x329a505 to 0x32a4c80, (0xa77b bytes) 2024-09-11T05:52:15.809 INFO:teuthology.orchestra.run.smithi165.stdout:4533 trunc from 0xa89bd00 to 0x4307f9f 2024-09-11T05:52:15.828 INFO:teuthology.orchestra.run.smithi165.stdout:4536 read 0xb9f232 thru 0xb9f35e (0x12d bytes) 2024-09-11T05:52:15.835 INFO:teuthology.orchestra.run.smithi165.stdout:4539 trunc from 0x4307f9f to 0x1f8192 2024-09-11T05:52:15.863 INFO:teuthology.orchestra.run.smithi165.stdout:4540 punch from 0x1bfa6e to 0x1ca88c, (0xae1e bytes) 2024-09-11T05:52:15.863 INFO:teuthology.orchestra.run.smithi165.stdout:4542 read 0x17c709 thru 0x188612 (0xbf0a bytes) 2024-09-11T05:52:15.887 INFO:teuthology.orchestra.run.smithi165.stdout:4543 write 0x9175c6a thru 0x9182fa9 (0xd340 bytes) 2024-09-11T05:52:15.895 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:15.892+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:15.897 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:15.897 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:15.899 INFO:teuthology.orchestra.run.smithi165.stdout:4546 writesame 0x12fe2c0 thru 0x1307de1 data_size 0x3(0x9b22 bytes) 2024-09-11T05:52:15.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:15.919 INFO:teuthology.orchestra.run.smithi165.stdout:4547 read 0x4b8453f thru 0x4b8c5b6 (0x8078 bytes) 2024-09-11T05:52:15.920 INFO:teuthology.orchestra.run.smithi165.stdout:4549 read 0x8c01f2f thru 0x8c0f56a (0xd63c bytes) 2024-09-11T05:52:15.920 INFO:teuthology.orchestra.run.smithi165.stdout:4552 write 0xf15ba1 thru 0xf15e5d (0x2bd bytes) 2024-09-11T05:52:15.923 INFO:teuthology.orchestra.run.smithi165.stdout:4553 read 0x43c4eb2 thru 0x43c5ff2 (0x1141 bytes) 2024-09-11T05:52:15.932 INFO:teuthology.orchestra.run.smithi165.stdout:4555 trunc from 0x9182faa to 0xdf7e528 2024-09-11T05:52:15.942 INFO:teuthology.orchestra.run.smithi165.stdout:4556 write 0xc6011c3 thru 0xc60e001 (0xce3f bytes) 2024-09-11T05:52:15.944 INFO:teuthology.orchestra.run.smithi165.stdout:4557 read 0x40cfc95 thru 0x40db2e3 (0xb64f bytes) 2024-09-11T05:52:15.944 INFO:teuthology.orchestra.run.smithi165.stdout:4558 clone 29 order 21 su 65536 sc 6 2024-09-11T05:52:15.997 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:15.991+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:16.048 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.048 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:16.098 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:16.096+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:16.199 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:16.195+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:16.238 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.238 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:16.300 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:16.295+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:16.400 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:16.395+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:16.435 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.435 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:16.537 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:16.535+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:16.602 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:16.599+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:16.619 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.619 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:52:16.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:16.703 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:16.700+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:16.803 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:16.799+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:16.804 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.804 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:16.904 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:16.904+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:16.979 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:16.979 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:16.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:17.005 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:17.004+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:17.106 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:17.103+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:17.109 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:17.109 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:17.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:17.209 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:17.207+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:17.284 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:17.284 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:17.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:17.342 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:17.339+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:17.409 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:17.403+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:17.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:17.466 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:17.466 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:17.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:17.509 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:17.508+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:17.588 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:17.588 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:17.605 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:17.605 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:17.610 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:17.608+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:17.622 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:17.622 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:17.622 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:52:17.653 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:17.653 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:17.655 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:17.655 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:17.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:17.676 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:17.676 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:17.695 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:17.695 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:17.711 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:17.707+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:17.715 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:17.715 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:17.734 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:17.734 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:17.791 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:17.791 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:17.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:17.812 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:17.811+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:17.913 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:17.911+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:17.971 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:17.971 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:17.972 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:17.972 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:17.972 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:17.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:18.014 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:18.015+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:18.105 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300298801, 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:18.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300271821999999, 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:18.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300241563, 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300196278000001, 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.119 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300163412, 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.124 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:18.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300135356999998, 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300108291000001, 2024-09-11T05:52:18.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300058289999999, 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:18.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:18.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.300025807000001, 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:18.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299997338000001, 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:18.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299968458999999, 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.151 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299911641000001, 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:18.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299880608999999, 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:18.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:18.163 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.29985259, 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.164 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299825661, 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:18.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.173 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299799095000001, 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:18.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:18.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:18.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:18.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299761883999999, 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:18.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:18.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299733330999999, 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:18.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:18.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.29968487, 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:18.186 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:18.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.299646254999999, 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:18.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.195 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:18.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:18.197 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:18.196+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:18.297 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:18.296+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:18.365 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:18.366 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:18.366 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:18.366 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:18.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:18.398 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:18.399+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:18.499 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:18.496+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:18.546 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:18.546 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:18.546 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:18.546 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:18.546 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:18.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:18.600 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:18.599+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:18.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:18.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:18.617 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 6, 7] 2024-09-11T05:52:18.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-11T05:52:18.700 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:18.700+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:18.736 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386513071000003, 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.737 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386483662000003, 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:18.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.745 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386463605000003, 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:18.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386422218, 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:18.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:18.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386389735999998, 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:18.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:18.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:18.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386369680000001, 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:18.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386350178000001, 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:18.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386325368999998, 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:18.767 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:18.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.771 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386306220000002, 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:18.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386285405999999, 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:18.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386265573000003, 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.780 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:18.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:18.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:18.784 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386245633000001, 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:18.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386219511999997, 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:18.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.386182927999997, 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.793 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:18.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:18.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.381395128999998, 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.797 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.381320381999998, 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.801 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:18.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:18.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.381247623, 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:18.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.381226703000003, 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.810 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:18.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.039260511999998, 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:18.814 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:18.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:18.820 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.330821101, 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.824 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:18.825 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:18.825 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:18.825 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone28 from 0xdf7e528 (overlap 0xee166) to 0x0 2024-09-11T05:52:18.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:18.827 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:18.824+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:18.880 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-09-11T05:52:18.927 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:18.927+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:18.974 INFO:tasks.ceph.mon.a.smithi165.stderr:2024-09-11T05:52:18.972+0000 7f2b6abd3640 -1 mon.a@0(leader).osd e85 definitely_dead 0 2024-09-11T05:52:18.997 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:18.997 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:18.997 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:18.997 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:19.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:19.056 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:19.056+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:19.129 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:19.128+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:19.174 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:19.174 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:19.174 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:19.174 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:19.174 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:19.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:19.233 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:19.232+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:19.293 INFO:teuthology.orchestra.run.smithi165.stdout:4559 read 0x6f246df thru 0x6f32260 (0xdb82 bytes) 2024-09-11T05:52:19.294 INFO:teuthology.orchestra.run.smithi165.stdout:4560 write 0x34e thru 0xce68 (0xcb1b bytes) 2024-09-11T05:52:19.312 INFO:teuthology.orchestra.run.smithi165.stdout:4563 write 0x845eec thru 0x84ae9b (0x4fb0 bytes) 2024-09-11T05:52:19.315 INFO:teuthology.orchestra.run.smithi165.stdout:4564 punch from 0xc728254 to 0xc7303e5, (0x8191 bytes) 2024-09-11T05:52:19.315 INFO:teuthology.orchestra.run.smithi165.stdout:4565 write 0xbe44553 thru 0xbe48e1a (0x48c8 bytes) 2024-09-11T05:52:19.323 INFO:teuthology.orchestra.run.smithi165.stdout:4567 write 0x85545d2 thru 0x855f6de (0xb10d bytes) 2024-09-11T05:52:19.330 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:19.328+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:19.332 INFO:teuthology.orchestra.run.smithi165.stdout:4568 punch from 0x7edff11 to 0x7ee7d6b, (0x7e5a bytes) 2024-09-11T05:52:19.333 INFO:teuthology.orchestra.run.smithi165.stdout:4569 read 0xa9afec7 thru 0xa9bd606 (0xd740 bytes) 2024-09-11T05:52:19.345 INFO:teuthology.orchestra.run.smithi165.stdout:4570 trunc from 0xdf7e528 to 0x4689541 2024-09-11T05:52:19.357 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:19.357 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:19.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:19.357 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.335574762999997, 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:19.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:19.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.844251565, 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.981274790999997, 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:19.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84925 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 114688~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.808720697999998, 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005558884, 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84925 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:19.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4000000000000002e-07 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547264+0000", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547271+0000", 2024-09-11T05:52:19.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547276+0000", 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7520000000000002e-06 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552413+0000", 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051366579999999997 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552501+0000", 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8220000000000006e-05 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552820+0000", 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031835400000000002 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.808697883000001, 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.808546364000001, 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.382 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:19.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.808503206000001, 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.390 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.808477382, 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:19.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:19.394 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.701004113, 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.395 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:19.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90109 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.569353358000001, 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055123259999999997, 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90109 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:19.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786632+0000", 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6969999999999999e-06 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786650+0000", 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8578999999999999e-05 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786697+0000", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786717+0000", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9579999999999999e-05 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786863+0000", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014619399999999999 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787123+0000", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026025400000000002 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787432+0000", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030917899999999998 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.790614+0000", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031820160000000002 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.791881+0000", 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001266585 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792030+0000", 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014938199999999999 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792140+0000", 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110152 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90695 3.0 3:03391bbf:::rbd_data.11242d2cfbf4.0000000000000079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.891723034000002, 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059067149999999999, 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90695 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464262+0000", 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:19.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464277+0000", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-05 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464325+0000", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8881000000000002e-05 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464351+0000", 2024-09-11T05:52:19.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5269000000000002e-05 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464527+0000", 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017588099999999999 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464774+0000", 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024752500000000001 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.465110+0000", 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033546800000000001 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.467391+0000", 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002281047 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470017+0000", 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026263079999999999 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470097+0000", 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0584e-05 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470165+0000", 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7655000000000002e-05 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:19.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.891580488999999, 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:19.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:19.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:19.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.545244994000001, 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:19.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:19.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:19.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:19.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.527256847, 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.425 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.429 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.465157562000002, 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.433 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.465045065999998, 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:19.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:19.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:19.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:19.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.985100778, 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:19.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:19.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:19.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.983526392, 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.443 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:19.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:19.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:19.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:19.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.448 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.450717349, 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:19.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:19.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:19.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:19.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.960429674, 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.454 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.460 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:19.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:19.462 INFO:teuthology.orchestra.run.smithi165.stdout:4571 writesame 0xe045e91 thru 0xe04a1f9 data_size 0x1(0x4369 bytes) 2024-09-11T05:52:19.462 INFO:teuthology.orchestra.run.smithi165.stdout:4573 trunc from 0xe04a1fa to 0x2efa20e 2024-09-11T05:52:19.462 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:19.463+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:19.492 INFO:teuthology.orchestra.run.smithi165.stdout:4575 read 0x24916f9 thru 0x249c1ee (0xaaf6 bytes) 2024-09-11T05:52:19.495 INFO:teuthology.orchestra.run.smithi165.stdout:4576 punch from 0x221c128 to 0x221d477, (0x134f bytes) 2024-09-11T05:52:19.495 INFO:teuthology.orchestra.run.smithi165.stdout:4577 trunc from 0x2efa20e to 0x2060683 2024-09-11T05:52:19.549 INFO:teuthology.orchestra.run.smithi165.stdout:4578 read 0x1cea75b thru 0x1cef0de (0x4984 bytes) 2024-09-11T05:52:19.563 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:19.560+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:19.578 INFO:teuthology.orchestra.run.smithi165.stdout:4579 write 0xedaf1c3 thru 0xedb1692 (0x24d0 bytes) 2024-09-11T05:52:19.586 INFO:teuthology.orchestra.run.smithi165.stdout:4580 read 0xe067d98 thru 0xe0776c7 (0xf930 bytes) 2024-09-11T05:52:19.586 INFO:teuthology.orchestra.run.smithi165.stdout:4581 read 0x3f35c9 thru 0x3ff3de (0xbe16 bytes) 2024-09-11T05:52:19.586 INFO:teuthology.orchestra.run.smithi165.stdout:4582 write 0xa3ee165 thru 0xa3f7043 (0x8edf bytes) 2024-09-11T05:52:19.596 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:19.596 INFO:teuthology.orchestra.run.smithi165.stdout:4586 write 0x17bc836 thru 0x17c37cf (0x6f9a bytes) 2024-09-11T05:52:19.608 INFO:teuthology.orchestra.run.smithi165.stdout:4587 writesame 0xdeba4eb thru 0xdec6184 data_size 0x27(0xbc9a bytes) 2024-09-11T05:52:19.620 INFO:teuthology.orchestra.run.smithi165.stdout:4588 punch from 0x3f8d0fc to 0x3f98402, (0xb306 bytes) 2024-09-11T05:52:19.620 INFO:teuthology.orchestra.run.smithi165.stdout:4590 read 0xe5aa6db thru 0xe5b086e (0x6194 bytes) 2024-09-11T05:52:19.620 INFO:teuthology.orchestra.run.smithi165.stdout:4591 read 0xb83fe50 thru 0xb848904 (0x8ab5 bytes) 2024-09-11T05:52:19.620 INFO:teuthology.orchestra.run.smithi165.stdout:4592 write 0x8dc19ee thru 0x8dcc2be (0xa8d1 bytes) 2024-09-11T05:52:19.626 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:19.626 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:19.626 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:19.626 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:19.627 INFO:teuthology.orchestra.run.smithi165.stdout:4593 write 0x6fe531b thru 0x6ff5297 (0xff7d bytes) 2024-09-11T05:52:19.636 INFO:teuthology.orchestra.run.smithi165.stderr:marked out osd.0. 2024-09-11T05:52:19.636 INFO:teuthology.orchestra.run.smithi165.stdout:4594 punch from 0x2edfd50 to 0x2ee6b0f, (0x6dbf bytes) 2024-09-11T05:52:19.637 INFO:teuthology.orchestra.run.smithi165.stdout:4595 read 0xcf4169a thru 0xcf43855 (0x21bc bytes) 2024-09-11T05:52:19.637 INFO:teuthology.orchestra.run.smithi165.stdout:4597 writesame 0x49c45af thru 0x49c4f8d data_size 0x7(0x9df bytes) 2024-09-11T05:52:19.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:19.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:52:19.663 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:19.660+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:19.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:19.764 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:19.763+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:19.813 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:19.814 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:19.814 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:19.814 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:19.814 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:19.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:19.866 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:19.867+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:19.899 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:19.899 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:19.932 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:19.932 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:19.963 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:19.963 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:19.966 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:19.964+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:19.986 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:19.986 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73336 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1331200~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913377187000002, 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034169613000000001, 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73336 2024-09-11T05:52:19.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071168+0000", 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071169+0000", 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4799999999999999e-07 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071170+0000", 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.821e-06 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071176+0000", 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8900000000000004e-06 2024-09-11T05:52:19.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071181+0000", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4469999999999999e-06 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104776+0000", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033594945000000001 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.104844+0000", 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8405999999999995e-05 2024-09-11T05:52:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105338+0000", 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049375600000000003 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73347 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1361920~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913358958000003, 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034640307000000002, 2024-09-11T05:52:19.991 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73347 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071187+0000", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071189+0000", 2024-09-11T05:52:19.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.843e-06 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071195+0000", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071199+0000", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6410000000000001e-06 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105705+0000", 2024-09-11T05:52:19.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034505909000000001 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105768+0000", 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2871000000000002e-05 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105827+0000", 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8795000000000002e-05 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:19.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73350 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1576960~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913340611000002, 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034310473000000001, 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73350 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:19.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:19.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5999999999999999e-07 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071207+0000", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.832e-06 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071213+0000", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7409999999999998e-06 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:19.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071217+0000", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4989999999999996e-06 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105375+0000", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034157869 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105440+0000", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5056999999999997e-05 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:19.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105515+0000", 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5115000000000002e-05 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73352 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1607680~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913322600000001, 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034453226000000003, 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73352 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071225+0000", 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071231+0000", 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8309999999999997e-06 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071235+0000", 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.476e-06 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105538+0000", 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034302323000000003 2024-09-11T05:52:20.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105603+0000", 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4969999999999996e-05 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105676+0000", 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3435000000000004e-05 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.004 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73353 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1581056~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913304353000001, 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035152165999999999, 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73353 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071243+0000", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071249+0000", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6869999999999999e-06 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071253+0000", 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106025+0000", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034771560999999999 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106088+0000", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2911999999999997e-05 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106393+0000", 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000305352 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73360 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 69632~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913286100999997, 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034953914000000003, 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73360 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.009 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071260+0000", 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7599999999999998e-07 2024-09-11T05:52:20.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071262+0000", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3769999999999999e-06 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071269+0000", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5130000000000002e-06 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071273+0000", 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8509999999999998e-06 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105850+0000", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034576140999999998 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105920+0000", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0351000000000005e-05 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106213+0000", 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000293305 2024-09-11T05:52:20.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73361 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 278528~18432] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913265529999997, 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035273718000000003, 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.013 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73361 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071282+0000", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071289+0000", 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3090000000000001e-06 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071293+0000", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7879999999999997e-06 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106234+0000", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034940183999999999 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106301+0000", 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7638000000000005e-05 2024-09-11T05:52:20.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106554+0000", 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025235999999999998 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73362 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 489472~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913245779999997, 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035455391000000003, 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73362 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0499999999999999e-07 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071302+0000", 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-06 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071308+0000", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071313+0000", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6299999999999997e-06 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106625+0000", 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035312398000000002 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106693+0000", 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7954999999999996e-05 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106755+0000", 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2055000000000001e-05 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73367 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1611776~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913227147999997, 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036019230999999999, 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73367 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071319+0000", 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4400000000000001e-07 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071325+0000", 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-11T05:52:20.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071331+0000", 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071335+0000", 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6319999999999999e-06 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106578+0000", 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035242348999999999 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107279+0000", 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070118999999999997 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107337+0000", 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8646000000000001e-05 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73370 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1888256~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913203199000002, 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035569649000000002, 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73370 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071343+0000", 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071344+0000", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071350+0000", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071355+0000", 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3850000000000004e-06 2024-09-11T05:52:20.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106778+0000", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035423221999999997 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106840+0000", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106912+0000", 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2181000000000006e-05 2024-09-11T05:52:20.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73372 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1329152~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913184913999999, 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035712460000000001, 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.037 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73372 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071361+0000", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9499999999999998e-07 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071363+0000", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071369+0000", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071374+0000", 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.78e-06 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106936+0000", 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035561622000000001 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106998+0000", 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1820000000000001e-05 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107073+0000", 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5210999999999995e-05 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73374 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1359872~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913165050000003, 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035851052000000001, 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73374 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:20.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071381+0000", 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5499999999999999e-07 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071383+0000", 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071388+0000", 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7370000000000003e-06 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071393+0000", 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8069999999999999e-06 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107095+0000", 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035701644999999997 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107157+0000", 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107231+0000", 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4828999999999998e-05 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.045 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73375 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1333248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913145964999998, 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036088839999999997, 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73375 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071400+0000", 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:52:20.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071402+0000", 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.148e-06 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071409+0000", 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5159999999999997e-06 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071414+0000", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0000000000000004e-06 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107365+0000", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035950332000000002 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107429+0000", 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4271999999999997e-05 2024-09-11T05:52:20.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107488+0000", 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9215999999999997e-05 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73377 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1363968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913124416000002, 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036963658000000003, 2024-09-11T05:52:20.050 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73377 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071424+0000", 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.419e-06 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071431+0000", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071437+0000", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0919999999999997e-06 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108262+0000", 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036825113999999999 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108323+0000", 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1292999999999997e-05 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108385+0000", 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1651999999999999e-05 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73380 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1579008~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.054 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913101996999998, 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036798605999999998, 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73380 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071444+0000", 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071446+0000", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2929999999999999e-06 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071453+0000", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0049999999999998e-06 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071458+0000", 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-06 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107511+0000", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036052646000000001 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108165+0000", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000653934 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108242+0000", 2024-09-11T05:52:20.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7222e-05 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73382 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1609728~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913080397999998, 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037077023000000001, 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73382 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.65e-07 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071468+0000", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071475+0000", 2024-09-11T05:52:20.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071480+0000", 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2020000000000003e-06 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108408+0000", 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036928043000000001 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108470+0000", 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1865999999999993e-05 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108542+0000", 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2206999999999995e-05 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73383 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1583104~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913058735999996, 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037204763000000002, 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73383 2024-09-11T05:52:20.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071489+0000", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.373e-06 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071495+0000", 2024-09-11T05:52:20.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8250000000000001e-06 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071500+0000", 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108564+0000", 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037063974 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108628+0000", 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3634999999999996e-05 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108691+0000", 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3647999999999998e-05 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.068 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73385 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1613824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913038743000001, 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037327601000000002, 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73385 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8299999999999998e-07 2024-09-11T05:52:20.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071509+0000", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0829999999999998e-06 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071517+0000", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071523+0000", 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9179999999999999e-06 2024-09-11T05:52:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108714+0000", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037191175999999999 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108775+0000", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108834+0000", 2024-09-11T05:52:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9447999999999997e-05 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73390 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1890304~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.913015903999998, 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037487624999999997, 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73390 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071530+0000", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0900000000000002e-07 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071532+0000", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071538+0000", 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071543+0000", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9930000000000003e-06 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108856+0000", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037312788 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108916+0000", 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9570000000000001e-05 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.109017+0000", 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101388 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.142903906000001, 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:20.082 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:20.083 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:20.083 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:20.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:20.084 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:20.084 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:20.085 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:20.085 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:20.085 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:20.085 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:20.085 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:20.086 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:20.086 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:20.083+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:20.184 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:20.184+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:20.218 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:20.218 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:20.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:20.285 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:20.283+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:20.349 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:20.349 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:20.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:20.386 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:20.384+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:20.487 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:20.484+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:20.506 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:52:20.537 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:20.537 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:20.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:20.598 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:20.598+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:20.669 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:20.670 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:20.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:20.689 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:20.688+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:20.790 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:20.790+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:20.848 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:20.848 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:20.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:20.906 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:20.906+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:20.992 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:20.990+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:21.032 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:21.032 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:21.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:21.130 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:21.130+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:21.193 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:21.190+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:21.209 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:21.210 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:21.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:21.294 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:21.292+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:21.391 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:21.391 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:21.394 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:21.394+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:21.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:21.498 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:21.498+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:21.579 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.579 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.580 INFO:teuthology.orchestra.run.smithi165.stdout:4601 punch from 0x85c78bb to 0x85cf39e, (0x7ae3 bytes) 2024-09-11T05:52:21.580 INFO:teuthology.orchestra.run.smithi165.stdout:4602 read 0x9661bd8 thru 0x966b3f6 (0x981f bytes) 2024-09-11T05:52:21.580 INFO:teuthology.orchestra.run.smithi165.stdout:4603 write 0x7719992 thru 0x7724409 (0xaa78 bytes) 2024-09-11T05:52:21.591 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:21.592 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:21.593 INFO:teuthology.orchestra.run.smithi165.stdout:4605 punch from 0xaec7efd to 0xaed0f9d, (0x90a0 bytes) 2024-09-11T05:52:21.593 INFO:teuthology.orchestra.run.smithi165.stdout:4606 write 0x1ae4f7a thru 0x1af34ef (0xe576 bytes) 2024-09-11T05:52:21.596 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:21.594+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:21.602 INFO:teuthology.orchestra.run.smithi165.stdout:4607 punch from 0x4f448cb to 0x4f4bdd4, (0x7509 bytes) 2024-09-11T05:52:21.602 INFO:teuthology.orchestra.run.smithi165.stdout:4608 write 0x7f7eec2 thru 0x7f887df (0x991e bytes) 2024-09-11T05:52:21.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:21.611 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.612 INFO:teuthology.orchestra.run.smithi165.stdout:4610 read 0x59d8fb2 thru 0x59e7738 (0xe787 bytes) 2024-09-11T05:52:21.612 INFO:teuthology.orchestra.run.smithi165.stdout:4611 punch from 0x8595330 to 0x85a25be, (0xd28e bytes) 2024-09-11T05:52:21.612 INFO:teuthology.orchestra.run.smithi165.stdout:4615 write 0x50ab519 thru 0x50ae939 (0x3421 bytes) 2024-09-11T05:52:21.618 INFO:teuthology.orchestra.run.smithi165.stdout:4616 writesame 0x2df4a52 thru 0x2dff22b data_size 0x5(0xa7da bytes) 2024-09-11T05:52:21.628 INFO:teuthology.orchestra.run.smithi165.stdout:4618 read 0xdaf82f9 thru 0xdb03245 (0xaf4d bytes) 2024-09-11T05:52:21.629 INFO:teuthology.orchestra.run.smithi165.stdout:4619 read 0x1a33e56 thru 0x1a37ece (0x4079 bytes) 2024-09-11T05:52:21.630 INFO:teuthology.orchestra.run.smithi165.stdout:4620 read 0x42cc44a thru 0x42d6944 (0xa4fb bytes) 2024-09-11T05:52:21.630 INFO:teuthology.orchestra.run.smithi165.stdout:4622 punch from 0x8a9f370 to 0x8aa27ec, (0x347c bytes) 2024-09-11T05:52:21.631 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.631 INFO:teuthology.orchestra.run.smithi165.stdout:4625 read 0x5d8421d thru 0x5d87382 (0x3166 bytes) 2024-09-11T05:52:21.631 INFO:teuthology.orchestra.run.smithi165.stdout:4626 write 0x2dc0247 thru 0x2dc0489 (0x243 bytes) 2024-09-11T05:52:21.634 INFO:teuthology.orchestra.run.smithi165.stdout:4627 write 0x443ad06 thru 0x4445887 (0xab82 bytes) 2024-09-11T05:52:21.649 INFO:teuthology.orchestra.run.smithi165.stdout:4628 read 0xc2c8327 thru 0xc2ce306 (0x5fe0 bytes) 2024-09-11T05:52:21.649 INFO:teuthology.orchestra.run.smithi165.stdout:4629 punch from 0x99c554d to 0x99d110b, (0xbbbe bytes) 2024-09-11T05:52:21.649 INFO:teuthology.orchestra.run.smithi165.stdout:4632 write 0x7a4e22d thru 0x7a50271 (0x2045 bytes) 2024-09-11T05:52:21.656 INFO:teuthology.orchestra.run.smithi165.stdout:4634 writesame 0xdbfef93 thru 0xdc001bc data_size 0x1f(0x122a bytes) 2024-09-11T05:52:21.664 INFO:teuthology.orchestra.run.smithi165.stdout:4635 write 0x837abc4 thru 0x837bac7 (0xf04 bytes) 2024-09-11T05:52:21.675 INFO:teuthology.orchestra.run.smithi165.stdout:4636 write 0x66f0e65 thru 0x66f9760 (0x88fc bytes) 2024-09-11T05:52:21.686 INFO:teuthology.orchestra.run.smithi165.stdout:4638 writesame 0x4862e87 thru 0x4864be7 data_size 0x3(0x1d61 bytes) 2024-09-11T05:52:21.692 INFO:teuthology.orchestra.run.smithi165.stdout:4640 punch from 0x18eaa17 to 0x18ebd45, (0x132e bytes) 2024-09-11T05:52:21.692 INFO:teuthology.orchestra.run.smithi165.stdout:4641 punch from 0x6fddaee to 0x6fe44fa, (0x6a0c bytes) 2024-09-11T05:52:21.693 INFO:teuthology.orchestra.run.smithi165.stdout:4642 write 0x202c1a6 thru 0x20315a5 (0x5400 bytes) 2024-09-11T05:52:21.698 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.698 INFO:teuthology.orchestra.run.smithi165.stdout:4644 writesame 0x3cfb143 thru 0x3cfdabe data_size 0x3c(0x297c bytes) 2024-09-11T05:52:21.701 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:21.702+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:21.708 INFO:teuthology.orchestra.run.smithi165.stdout:4645 write 0x5656192 thru 0x5658e90 (0x2cff bytes) 2024-09-11T05:52:21.714 INFO:teuthology.orchestra.run.smithi165.stdout:4647 read 0x9754df8 thru 0x976339c (0xe5a5 bytes) 2024-09-11T05:52:21.714 INFO:teuthology.orchestra.run.smithi165.stdout:4648 write 0xe652bb6 thru 0xe662935 (0xfd80 bytes) 2024-09-11T05:52:21.730 INFO:teuthology.orchestra.run.smithi165.stdout:4649 read 0xc962111 thru 0xc9679f4 (0x58e4 bytes) 2024-09-11T05:52:21.731 INFO:teuthology.orchestra.run.smithi165.stdout:4653 writesame 0x68230e0 thru 0x682dae2 data_size 0x1(0xaa03 bytes) 2024-09-11T05:52:21.763 INFO:teuthology.orchestra.run.smithi165.stdout:4654 punch from 0xe2cde83 to 0xe2d8fb9, (0xb136 bytes) 2024-09-11T05:52:21.763 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.763 INFO:teuthology.orchestra.run.smithi165.stdout:4656 read 0xce55cb6 thru 0xce5c858 (0x6ba3 bytes) 2024-09-11T05:52:21.763 INFO:teuthology.orchestra.run.smithi165.stdout:4657 write 0x6f3312e thru 0x6f347bb (0x168e bytes) 2024-09-11T05:52:21.765 INFO:teuthology.orchestra.run.smithi165.stdout:4658 write 0xa0f7422 thru 0xa0fa4e3 (0x30c2 bytes) 2024-09-11T05:52:21.790 INFO:teuthology.orchestra.run.smithi165.stdout:4660 trunc from 0xedb1693 to 0x41b2778 2024-09-11T05:52:21.797 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:21.794+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:21.811 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:21.811 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:21.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:21.830 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.830 INFO:teuthology.orchestra.run.smithi165.stdout:4662 read 0x293d257 thru 0x294b6a3 (0xe44d bytes) 2024-09-11T05:52:21.831 INFO:teuthology.orchestra.run.smithi165.stdout:4664 read 0x29ff28e thru 0x29ffa4a (0x7bd bytes) 2024-09-11T05:52:21.831 INFO:teuthology.orchestra.run.smithi165.stdout:4665 read 0x17fe021 thru 0x180038e (0x236e bytes) 2024-09-11T05:52:21.831 INFO:teuthology.orchestra.run.smithi165.stdout:4666 writesame 0xd8f228 thru 0xd91c41 data_size 0x2(0x2a1a bytes) 2024-09-11T05:52:21.867 INFO:teuthology.orchestra.run.smithi165.stdout:4667 write 0xdaaa156 thru 0xdaac991 (0x283c bytes) 2024-09-11T05:52:21.872 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.872 INFO:teuthology.orchestra.run.smithi165.stdout:4669 read 0x72e2369 thru 0x72e8acb (0x6763 bytes) 2024-09-11T05:52:21.872 INFO:teuthology.orchestra.run.smithi165.stdout:4671 read 0xd2529d thru 0xd26031 (0xd95 bytes) 2024-09-11T05:52:21.872 INFO:teuthology.orchestra.run.smithi165.stdout:4672 writesame 0x3972893 thru 0x397f580 data_size 0x11(0xccee bytes) 2024-09-11T05:52:21.879 INFO:teuthology.orchestra.run.smithi165.stdout:4673 write 0xcab2d1e thru 0xcab7571 (0x4854 bytes) 2024-09-11T05:52:21.889 INFO:teuthology.orchestra.run.smithi165.stdout:4674 punch from 0x3d50ca to 0x3e075c, (0xb692 bytes) 2024-09-11T05:52:21.889 INFO:teuthology.orchestra.run.smithi165.stdout:4675 write 0x3e135c0 thru 0x3e1fe71 (0xc8b2 bytes) 2024-09-11T05:52:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:4677 punch from 0x3f5b35c to 0x3f6aa1f, (0xf6c3 bytes) 2024-09-11T05:52:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:4678 punch from 0x3864991 to 0x386be92, (0x7501 bytes) 2024-09-11T05:52:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:4680 punch from 0xcbe32f8 to 0xcbf00a7, (0xcdaf bytes) 2024-09-11T05:52:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:4681 write 0x8bb0518 thru 0x8bc0357 (0xfe40 bytes) 2024-09-11T05:52:21.899 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:21.896+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:21.908 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:21.908 INFO:teuthology.orchestra.run.smithi165.stdout:4686 write 0x8522a56 thru 0x852e1b4 (0xb75f bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4687 read 0x89334a4 thru 0x893b4d5 (0x8032 bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4688 punch from 0x3eacdd2 to 0x3eb43a3, (0x75d1 bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4689 read 0xcc9df7b thru 0xccac08a (0xe110 bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4690 punch from 0x5910634 to 0x591b1e3, (0xabaf bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4691 read 0x4604ebd thru 0x460aeb4 (0x5ff8 bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4693 punch from 0x94d65e3 to 0x94e6565, (0xff82 bytes) 2024-09-11T05:52:21.917 INFO:teuthology.orchestra.run.smithi165.stdout:4694 writesame 0x87e689a thru 0x87f5004 data_size 0x1(0xe76b bytes) 2024-09-11T05:52:21.986 INFO:teuthology.orchestra.run.smithi165.stdout:4699 trunc from 0xdaac992 to 0x555301f 2024-09-11T05:52:22.000 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:22.000+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:22.001 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:22.002 INFO:teuthology.orchestra.run.smithi165.stdout:4701 read 0x3d6e70a thru 0x3d71f84 (0x387b bytes) 2024-09-11T05:52:22.002 INFO:teuthology.orchestra.run.smithi165.stdout:4702 read 0x2ccdc1b thru 0x2cd5220 (0x7606 bytes) 2024-09-11T05:52:22.012 INFO:teuthology.orchestra.run.smithi165.stdout:4703 write 0x5f38e16 thru 0x5f4008c (0x7277 bytes) 2024-09-11T05:52:22.013 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:22.013 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:22.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:22.041 INFO:teuthology.orchestra.run.smithi165.stdout:4704 write 0xb588d94 thru 0xb596f30 (0xe19d bytes) 2024-09-11T05:52:22.047 INFO:teuthology.orchestra.run.smithi165.stdout:4705 write 0x4100038 thru 0x410905f (0x9028 bytes) 2024-09-11T05:52:22.056 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:22.056 INFO:teuthology.orchestra.run.smithi165.stdout:4707 write 0xaef683 thru 0xaf6073 (0x69f1 bytes) 2024-09-11T05:52:22.065 INFO:teuthology.orchestra.run.smithi165.stdout:4708 write 0x96bea93 thru 0x96c22df (0x384d bytes) 2024-09-11T05:52:22.076 INFO:teuthology.orchestra.run.smithi165.stdout:4711 trunc from 0xb596f31 to 0xd4ed8b1 2024-09-11T05:52:22.084 INFO:teuthology.orchestra.run.smithi165.stdout:4712 read 0x6d7e3af thru 0x6d80f57 (0x2ba9 bytes) 2024-09-11T05:52:22.084 INFO:teuthology.orchestra.run.smithi165.stdout:4714 write 0x9dd636a thru 0x9de0679 (0xa310 bytes) 2024-09-11T05:52:22.089 INFO:teuthology.orchestra.run.smithi165.stdout:4715 writesame 0xe45930 thru 0xe4fe64 data_size 0x1(0xa535 bytes) 2024-09-11T05:52:22.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:22.100 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:22.098+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:22.113 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:22.113 INFO:teuthology.orchestra.run.smithi165.stdout:4718 write 0x6859b20 thru 0x6860cb2 (0x7193 bytes) 2024-09-11T05:52:22.122 INFO:teuthology.orchestra.run.smithi165.stdout:4719 read 0xc911340 thru 0xc9139a7 (0x2668 bytes) 2024-09-11T05:52:22.122 INFO:teuthology.orchestra.run.smithi165.stdout:4720 write 0x6c6ad90 thru 0x6c6e5c4 (0x3835 bytes) 2024-09-11T05:52:22.126 INFO:teuthology.orchestra.run.smithi165.stdout:4721 writesame 0xc75c128 thru 0xc760627 data_size 0x18(0x4500 bytes) 2024-09-11T05:52:22.133 INFO:teuthology.orchestra.run.smithi165.stdout:4722 read 0x4bb4682 thru 0x4bb6b73 (0x24f2 bytes) 2024-09-11T05:52:22.133 INFO:teuthology.orchestra.run.smithi165.stdout:4725 read 0x5edb2d0 thru 0x5edd0b5 (0x1de6 bytes) 2024-09-11T05:52:22.133 INFO:teuthology.orchestra.run.smithi165.stdout:4726 read 0xafe9d9d thru 0xafec38b (0x25ef bytes) 2024-09-11T05:52:22.134 INFO:teuthology.orchestra.run.smithi165.stdout:4727 punch from 0xd3ba64e to 0xd3bce1f, (0x27d1 bytes) 2024-09-11T05:52:22.134 INFO:teuthology.orchestra.run.smithi165.stdout:4728 flatten 2024-09-11T05:52:22.162 INFO:teuthology.orchestra.run.smithi165.stdout:4731 read 0x6a0f906 thru 0x6a16b0b (0x7206 bytes) 2024-09-11T05:52:22.162 INFO:teuthology.orchestra.run.smithi165.stdout:4732 clone 30 order 24 su 262144 sc 2 2024-09-11T05:52:22.166 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:22.166 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:22.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:22.232 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:22.232+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:22.263 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:22.263 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:22.281 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:22.281 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:22.299 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:22.299 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:22.301 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:22.298+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:22.306 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:22.306 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:22.306 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:22.306 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:22.318 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:22.318 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:22.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:22.342 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:22.343 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:22.366 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:22.366 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:22.389 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:22.389 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:22.402 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:22.402+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:22.411 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:22.412 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:22.469 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:22.469 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:22.469 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:22.469 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:22.469 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:22.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:22.532 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:22.532+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:22.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:52:22.604 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:22.602+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.653 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90356 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10076160~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849270348000001, 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095820444000000005, 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90356 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802756+0000", 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802757+0000", 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802760+0000", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6189999999999998e-06 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802768+0000", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3070000000000003e-06 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802775+0000", 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8499999999999996e-06 2024-09-11T05:52:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898288+0000", 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095513211000000001 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898421+0000", 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132526 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898577+0000", 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000156324 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849243369, 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.658 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:22.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.662 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849213110000001, 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:22.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:22.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:22.666 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849167824999999, 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.667 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:22.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849134959000001, 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.672 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849106903999999, 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.676 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.680 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849079838000002, 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:22.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:22.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.849029837, 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.685 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:22.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848997354000002, 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:22.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:22.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848968885000001, 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:22.693 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848940005999999, 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:22.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:22.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:22.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848883187999999, 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.702 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:22.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:22.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848852156, 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.707 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848824137000001, 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.715 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848797208000001, 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:22.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.719 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848770642000002, 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.720 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:22.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:22.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848733430999999, 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:22.724 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.728 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848704877999999, 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848656417000001, 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.848617802, 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:22.737 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:22.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:22.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:22.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:22.741 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:22.742 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:22.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:22.743 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:22.742+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:22.844 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:22.842+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:22.859 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:22.859 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:22.859 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:22.859 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:22.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:22.945 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:22.942+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:22.988 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:22.988 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:22.988 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:23.046 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:23.046+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:23.127 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:23.127 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777196308000001, 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:23.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:23.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777166899000001, 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777146842000001, 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:23.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:23.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777105454999997, 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:23.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:23.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:23.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777072973000003, 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.145 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:23.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777052916999999, 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777033414999998, 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.157 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.777008606000003, 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:23.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:23.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.776989456999999, 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:23.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:23.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.776968643000004, 2024-09-11T05:52:23.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:23.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.170 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.77694881, 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.171 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:23.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:23.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:23.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.776928869999999, 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:23.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:23.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.179 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.776902749000001, 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:23.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:23.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.776866165000001, 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.772078366000002, 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:23.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:23.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.772003619000003, 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:23.192 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.771930859999998, 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.77190994, 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:23.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.429943749000003, 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.205 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.721504337999999, 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:23.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.215 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:23.217 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:23.216+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:23.318 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:23.318+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:23.387 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:23.387 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:23.387 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:23.387 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:23.419 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:23.418+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:23.519 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:23.518+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:23.520 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:52:23.530 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:23.531 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:23.531 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:23.531 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:23.531 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:23.620 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:23.620+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:23.688 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.666568175000002, 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:23.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:23.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.175244976999998, 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.693 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:23.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:23.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.312268203000002, 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:23.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.703 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84925 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 114688~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.13971411, 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005558884, 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84925 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547261+0000", 2024-09-11T05:52:23.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4000000000000002e-07 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547264+0000", 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547271+0000", 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6089999999999997e-06 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547276+0000", 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7520000000000002e-06 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552413+0000", 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051366579999999997 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552501+0000", 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8220000000000006e-05 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552820+0000", 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031835400000000002 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.139691294999999, 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:23.708 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:23.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:23.712 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.139539775999999, 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.713 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:23.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.717 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.139496618000003, 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:23.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:23.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.139470793999998, 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.722 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:23.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.031997525000001, 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:23.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:23.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90109 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.900346769999999, 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055123259999999997, 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90109 2024-09-11T05:52:23.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786628+0000", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786632+0000", 2024-09-11T05:52:23.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6969999999999999e-06 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786650+0000", 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8578999999999999e-05 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786697+0000", 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786717+0000", 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9579999999999999e-05 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.786863+0000", 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014619399999999999 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787123+0000", 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026025400000000002 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.787432+0000", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030917899999999998 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.790614+0000", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031820160000000002 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.791881+0000", 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001266585 2024-09-11T05:52:23.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792030+0000", 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014938199999999999 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.792140+0000", 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110152 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90695 3.0 3:03391bbf:::rbd_data.11242d2cfbf4.0000000000000079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.222716446, 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059067149999999999, 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90695 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.739 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464262+0000", 2024-09-11T05:52:23.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464277+0000", 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-05 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464325+0000", 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8881000000000002e-05 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464351+0000", 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5269000000000002e-05 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464527+0000", 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017588099999999999 2024-09-11T05:52:23.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464774+0000", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024752500000000001 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.465110+0000", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033546800000000001 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.467391+0000", 2024-09-11T05:52:23.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002281047 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470017+0000", 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026263079999999999 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470097+0000", 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0584e-05 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470165+0000", 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7655000000000002e-05 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.222573901000001, 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.745 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:23.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.876238405999999, 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.751 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:23.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:23.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:23.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.858250258999998, 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:23.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:23.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:23.761 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.796150974, 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:23.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:23.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.796038478, 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:23.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:23.770 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.316094190000001, 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:23.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.314519804, 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:23.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.781710760999999, 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:23.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.291423086000002, 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.789 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:23.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:23.795 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.796 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.796 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.796 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.796 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:23.798 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:23.794+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:23.901 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:23.900+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3.3fc69617 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.22991156600000001, 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.22993359899999999, 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for peered", 2024-09-11T05:52:23.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:23.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:52:23.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-11T05:52:23.962 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:23.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:24.002 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:24.002+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:24.104 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:24.104+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:24.134 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:24.134 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:24.134 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:24.134 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:24.134 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:24.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:24.205 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:24.202+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:24.306 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:24.304+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:24.317 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73347 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1361920~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244359299999999, 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034640307000000002, 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73347 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071187+0000", 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071189+0000", 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.843e-06 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071195+0000", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071199+0000", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6410000000000001e-06 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105705+0000", 2024-09-11T05:52:24.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034505909000000001 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105768+0000", 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2871000000000002e-05 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105827+0000", 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8795000000000002e-05 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73350 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1576960~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244340952999998, 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034310473000000001, 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73350 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.323 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071205+0000", 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5999999999999999e-07 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071207+0000", 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.832e-06 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071213+0000", 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7409999999999998e-06 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071217+0000", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4989999999999996e-06 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105375+0000", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034157869 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105440+0000", 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5056999999999997e-05 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105515+0000", 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5115000000000002e-05 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73352 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1607680~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244322941999997, 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034453226000000003, 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73352 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:24.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071225+0000", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071231+0000", 2024-09-11T05:52:24.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8309999999999997e-06 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071235+0000", 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.476e-06 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105538+0000", 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034302323000000003 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105603+0000", 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4969999999999996e-05 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105676+0000", 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3435000000000004e-05 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73353 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1581056~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:24.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244304694999997, 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035152165999999999, 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73353 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071243+0000", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071249+0000", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6869999999999999e-06 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071253+0000", 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106025+0000", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034771560999999999 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106088+0000", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2911999999999997e-05 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106393+0000", 2024-09-11T05:52:24.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000305352 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73360 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 69632~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244286443, 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034953914000000003, 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73360 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071260+0000", 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7599999999999998e-07 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071262+0000", 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3769999999999999e-06 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071269+0000", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5130000000000002e-06 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071273+0000", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8509999999999998e-06 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105850+0000", 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034576140999999998 2024-09-11T05:52:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105920+0000", 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0351000000000005e-05 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106213+0000", 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000293305 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73361 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 278528~18432] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244265872, 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035273718000000003, 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73361 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071282+0000", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071289+0000", 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3090000000000001e-06 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071293+0000", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7879999999999997e-06 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106234+0000", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034940183999999999 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106301+0000", 2024-09-11T05:52:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7638000000000005e-05 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106554+0000", 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025235999999999998 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73362 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 489472~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244246122, 2024-09-11T05:52:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035455391000000003, 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73362 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0499999999999999e-07 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071302+0000", 2024-09-11T05:52:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-06 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071308+0000", 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071313+0000", 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6299999999999997e-06 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106625+0000", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035312398000000002 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106693+0000", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7954999999999996e-05 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106755+0000", 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2055000000000001e-05 2024-09-11T05:52:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73367 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1611776~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.24422749, 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036019230999999999, 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73367 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071319+0000", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4400000000000001e-07 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071325+0000", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071331+0000", 2024-09-11T05:52:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071335+0000", 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6319999999999999e-06 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106578+0000", 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035242348999999999 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107279+0000", 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070118999999999997 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107337+0000", 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8646000000000001e-05 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73370 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1888256~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244203540999997, 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035569649000000002, 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73370 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071343+0000", 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071344+0000", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071350+0000", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071355+0000", 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3850000000000004e-06 2024-09-11T05:52:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106778+0000", 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035423221999999997 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106840+0000", 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106912+0000", 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2181000000000006e-05 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73372 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1329152~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244185256000002, 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035712460000000001, 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73372 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071361+0000", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9499999999999998e-07 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071363+0000", 2024-09-11T05:52:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071369+0000", 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071374+0000", 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.78e-06 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106936+0000", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035561622000000001 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106998+0000", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1820000000000001e-05 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107073+0000", 2024-09-11T05:52:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5210999999999995e-05 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73374 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1359872~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244165391999999, 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035851052000000001, 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73374 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071381+0000", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5499999999999999e-07 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071383+0000", 2024-09-11T05:52:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071388+0000", 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7370000000000003e-06 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071393+0000", 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8069999999999999e-06 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107095+0000", 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035701644999999997 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107157+0000", 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107231+0000", 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4828999999999998e-05 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73375 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1333248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244146307000001, 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036088839999999997, 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73375 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071400+0000", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071402+0000", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.148e-06 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071409+0000", 2024-09-11T05:52:24.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5159999999999997e-06 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071414+0000", 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0000000000000004e-06 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107365+0000", 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035950332000000002 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107429+0000", 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4271999999999997e-05 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107488+0000", 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9215999999999997e-05 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73377 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1363968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244124757999998, 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036963658000000003, 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73377 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071424+0000", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.419e-06 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071431+0000", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071437+0000", 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0919999999999997e-06 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108262+0000", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036825113999999999 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108323+0000", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1292999999999997e-05 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108385+0000", 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1651999999999999e-05 2024-09-11T05:52:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73380 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1579008~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244102339000001, 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036798605999999998, 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73380 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071444+0000", 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071446+0000", 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2929999999999999e-06 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071453+0000", 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0049999999999998e-06 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071458+0000", 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-06 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107511+0000", 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036052646000000001 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108165+0000", 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000653934 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108242+0000", 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7222e-05 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73382 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1609728~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244080740000001, 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037077023000000001, 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73382 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.65e-07 2024-09-11T05:52:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071468+0000", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071475+0000", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071480+0000", 2024-09-11T05:52:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2020000000000003e-06 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108408+0000", 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036928043000000001 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108470+0000", 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1865999999999993e-05 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108542+0000", 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2206999999999995e-05 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73383 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1583104~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244059077999999, 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037204763000000002, 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73383 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071489+0000", 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.373e-06 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071495+0000", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8250000000000001e-06 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071500+0000", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108564+0000", 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037063974 2024-09-11T05:52:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108628+0000", 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3634999999999996e-05 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108691+0000", 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3647999999999998e-05 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73385 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1613824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244039084999997, 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037327601000000002, 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73385 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8299999999999998e-07 2024-09-11T05:52:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071509+0000", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0829999999999998e-06 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071517+0000", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071523+0000", 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9179999999999999e-06 2024-09-11T05:52:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108714+0000", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037191175999999999 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108775+0000", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108834+0000", 2024-09-11T05:52:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9447999999999997e-05 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73390 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1890304~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.244016246000001, 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037487624999999997, 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73390 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071530+0000", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0900000000000002e-07 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071532+0000", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071538+0000", 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071543+0000", 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9930000000000003e-06 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108856+0000", 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037312788 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108916+0000", 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9570000000000001e-05 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.109017+0000", 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101388 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.473904247999997, 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.741287791, 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:52:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:24.412 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:24.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:24.414 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:24.414+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:24.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:24.514 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:24.514+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:24.549 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:24.550 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:24.550 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:24.554 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-09-11T05:52:24.557 INFO:teuthology.orchestra.run.smithi165.stdout:4733 read 0x5d1f396 thru 0x5d2b941 (0xc5ac bytes) 2024-09-11T05:52:24.557 INFO:teuthology.orchestra.run.smithi165.stdout:4735 writesame 0x4d69e1d thru 0x4d78315 data_size 0x27(0xe4f9 bytes) 2024-09-11T05:52:24.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:24.577 INFO:teuthology.orchestra.run.smithi165.stdout:4737 write 0xd1afc2 thru 0xd28601 (0xd640 bytes) 2024-09-11T05:52:24.590 INFO:teuthology.orchestra.run.smithi165.stdout:4739 writesame 0x47813ea thru 0x4782625 data_size 0x6(0x123c bytes) 2024-09-11T05:52:24.599 INFO:teuthology.orchestra.run.smithi165.stdout:4742 punch from 0x12cff16 to 0x12dba19, (0xbb03 bytes) 2024-09-11T05:52:24.599 INFO:teuthology.orchestra.run.smithi165.stdout:4743 read 0x8f292cc thru 0x8f33a3a (0xa76f bytes) 2024-09-11T05:52:24.615 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:24.615 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:24.616 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:24.612+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:24.616 INFO:teuthology.orchestra.run.smithi165.stdout:4744 trunc from 0xd4ed8b1 to 0x1d8017c 2024-09-11T05:52:24.637 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:24.637 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:24.664 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:24.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:24.664 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T05:52:24.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T05:52:24.665 INFO:teuthology.orchestra.run.smithi165.stdout:4745 writesame 0xb0daef8 thru 0xb0e2b1e data_size 0x1(0x7c27 bytes) 2024-09-11T05:52:24.665 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.666 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.666 INFO:teuthology.orchestra.run.smithi165.stdout:4750 writesame 0x6df01b1 thru 0x6dff3a5 data_size 0xb(0xf1f5 bytes) 2024-09-11T05:52:24.666 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:24.666 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:24.674 INFO:teuthology.orchestra.run.smithi165.stdout:4752 trunc from 0xb0e2b1f to 0x9f6fba0 2024-09-11T05:52:24.681 INFO:teuthology.orchestra.run.smithi165.stdout:4753 write 0x848d2e8 thru 0x84985b0 (0xb2c9 bytes) 2024-09-11T05:52:24.684 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:24.685 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:24.686 INFO:teuthology.orchestra.run.smithi165.stdout:4754 write 0x37cac92 thru 0x37d2ecf (0x823e bytes) 2024-09-11T05:52:24.691 INFO:teuthology.orchestra.run.smithi165.stdout:4755 read 0x9e53e72 thru 0x9e62bce (0xed5d bytes) 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:4757 punch from 0x782c489 to 0x7831272, (0x4de9 bytes) 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:4759 read 0x4eeadde thru 0x4ef9061 (0xe284 bytes) 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:4760 read 0x93ede19 thru 0x93f3ec8 (0x60b0 bytes) 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:4766 punch from 0x6f6ec9f to 0x6f756b1, (0x6a12 bytes) 2024-09-11T05:52:24.692 INFO:teuthology.orchestra.run.smithi165.stdout:4768 write 0x7ee4c6b thru 0x7eee2ff (0x9695 bytes) 2024-09-11T05:52:24.694 INFO:teuthology.orchestra.run.smithi165.stdout:4769 writesame 0x6197ff9 thru 0x61a195d data_size 0x1(0x9965 bytes) 2024-09-11T05:52:24.707 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:24.708 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:24.716 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:24.716+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:24.727 INFO:teuthology.orchestra.run.smithi165.stdout:4770 writesame 0xcbd78ca thru 0xcbe0d42 data_size 0x1(0x9479 bytes) 2024-09-11T05:52:24.730 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:24.730 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:24.731 INFO:teuthology.orchestra.run.smithi165.stdout:4771 trunc from 0xcbe0d43 to 0xa2daa55 2024-09-11T05:52:24.736 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:24.736 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:24.738 INFO:teuthology.orchestra.run.smithi165.stdout:4773 write 0x1913df4 thru 0x191704d (0x325a bytes) 2024-09-11T05:52:24.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:24.752 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:24.752 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:24.757 INFO:teuthology.orchestra.run.smithi165.stdout:4774 writesame 0xec6e0da thru 0xec7cd8c data_size 0x1(0xecb3 bytes) 2024-09-11T05:52:24.762 INFO:teuthology.orchestra.run.smithi165.stdout:4775 trunc from 0xec7cd8d to 0xb437b01 2024-09-11T05:52:24.768 INFO:teuthology.orchestra.run.smithi165.stdout:4776 read 0x5af72ce thru 0x5afda36 (0x6769 bytes) 2024-09-11T05:52:24.768 INFO:teuthology.orchestra.run.smithi165.stdout:4777 write 0x4a40d3c thru 0x4a4bf4d (0xb212 bytes) 2024-09-11T05:52:24.773 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:24.773 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:24.775 INFO:teuthology.orchestra.run.smithi165.stdout:4779 read 0x3ae55d1 thru 0x3aeabad (0x55dd bytes) 2024-09-11T05:52:24.776 INFO:teuthology.orchestra.run.smithi165.stdout:4780 read 0x3d1d66a thru 0x3d26635 (0x8fcc bytes) 2024-09-11T05:52:24.776 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.776 INFO:teuthology.orchestra.run.smithi165.stdout:4782 write 0x8891acc thru 0x889fc94 (0xe1c9 bytes) 2024-09-11T05:52:24.778 INFO:teuthology.orchestra.run.smithi165.stdout:4783 read 0x3fbdfe6 thru 0x3fc3f48 (0x5f63 bytes) 2024-09-11T05:52:24.778 INFO:teuthology.orchestra.run.smithi165.stdout:4785 write 0x3d8e17c thru 0x3d915a1 (0x3426 bytes) 2024-09-11T05:52:24.782 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.782 INFO:teuthology.orchestra.run.smithi165.stdout:4787 read 0x192f98c thru 0x193e599 (0xec0e bytes) 2024-09-11T05:52:24.792 INFO:teuthology.orchestra.run.smithi165.stdout:4789 trunc from 0xb437b01 to 0x49e35dc 2024-09-11T05:52:24.821 INFO:teuthology.orchestra.run.smithi165.stdout:4790 writesame 0x92724b0 thru 0x927ca29 data_size 0x3b(0xa57a bytes) 2024-09-11T05:52:24.822 INFO:teuthology.orchestra.run.smithi165.stdout:4791 read 0x5452f66 thru 0x545e656 (0xb6f1 bytes) 2024-09-11T05:52:24.822 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:24.820+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:24.822 INFO:teuthology.orchestra.run.smithi165.stdout:4792 read 0x2779d37 thru 0x2785aa7 (0xbd71 bytes) 2024-09-11T05:52:24.823 INFO:teuthology.orchestra.run.smithi165.stdout:4793 write 0xae5808 thru 0xaed3ef (0x7be8 bytes) 2024-09-11T05:52:24.825 INFO:teuthology.orchestra.run.smithi165.stdout:4794 read 0x853bb35 thru 0x8546247 (0xa713 bytes) 2024-09-11T05:52:24.825 INFO:teuthology.orchestra.run.smithi165.stdout:4795 write 0xa20f6d thru 0xa26b9b (0x5c2f bytes) 2024-09-11T05:52:24.832 INFO:teuthology.orchestra.run.smithi165.stdout:4796 trunc from 0x927ca2a to 0x5f0af3d 2024-09-11T05:52:24.844 INFO:teuthology.orchestra.run.smithi165.stdout:4797 punch from 0x569c50c to 0x56a5cb4, (0x97a8 bytes) 2024-09-11T05:52:24.858 INFO:teuthology.orchestra.run.smithi165.stdout:4798 trunc from 0x5f0af3d to 0xe14ae04 2024-09-11T05:52:24.868 INFO:teuthology.orchestra.run.smithi165.stdout:4799 trunc from 0xe14ae04 to 0xc3c0174 2024-09-11T05:52:24.874 INFO:teuthology.orchestra.run.smithi165.stdout:4800 write 0x838fc8d thru 0x8392769 (0x2add bytes) 2024-09-11T05:52:24.876 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:24.876 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:24.880 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.880 INFO:teuthology.orchestra.run.smithi165.stdout:4802 punch from 0x1e6ec55 to 0x1e762f4, (0x769f bytes) 2024-09-11T05:52:24.880 INFO:teuthology.orchestra.run.smithi165.stdout:4803 writesame 0x874e110 thru 0x8753399 data_size 0x5(0x528a bytes) 2024-09-11T05:52:24.885 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:24.903 INFO:teuthology.orchestra.run.smithi165.stdout:4806 trunc from 0xc3c0174 to 0x23ed2ac 2024-09-11T05:52:24.921 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.921 INFO:teuthology.orchestra.run.smithi165.stdout:4809 write 0x1d12e87 thru 0x1d1789a (0x4a14 bytes) 2024-09-11T05:52:24.926 INFO:teuthology.orchestra.run.smithi165.stdout:4810 trunc from 0x23ed2ac to 0x1263591 2024-09-11T05:52:24.937 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:24.938+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:24.947 INFO:teuthology.orchestra.run.smithi165.stdout:4811 punch from 0xe8dd33 to 0xe90cbb, (0x2f88 bytes) 2024-09-11T05:52:24.947 INFO:teuthology.orchestra.run.smithi165.stdout:4812 read 0xdc63a9 thru 0xdceb7b (0x87d3 bytes) 2024-09-11T05:52:24.948 INFO:teuthology.orchestra.run.smithi165.stdout:4813 punch from 0xe24805 to 0xe27551, (0x2d4c bytes) 2024-09-11T05:52:24.948 INFO:teuthology.orchestra.run.smithi165.stdout:4814 punch from 0xdf771d to 0xdf8f76, (0x1859 bytes) 2024-09-11T05:52:24.948 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.948 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:24.972 INFO:teuthology.orchestra.run.smithi165.stdout:4817 writesame 0x9880307 thru 0x9889bb7 data_size 0x1(0x98b1 bytes) 2024-09-11T05:52:24.972 INFO:teuthology.orchestra.run.smithi165.stdout:4819 read 0x197e62a thru 0x198396f (0x5346 bytes) 2024-09-11T05:52:24.988 INFO:teuthology.orchestra.run.smithi165.stdout:4821 write 0xb69413d thru 0xb694ea1 (0xd65 bytes) 2024-09-11T05:52:25.005 INFO:teuthology.orchestra.run.smithi165.stdout:4822 write 0xc1a3847 thru 0xc1aba66 (0x8220 bytes) 2024-09-11T05:52:25.012 INFO:teuthology.orchestra.run.smithi165.stdout:4823 write 0x4e38389 thru 0x4e3d39c (0x5014 bytes) 2024-09-11T05:52:25.018 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.018 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.018 INFO:teuthology.orchestra.run.smithi165.stdout:4824 write 0x593aaea thru 0x5948655 (0xdb6c bytes) 2024-09-11T05:52:25.019 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:25.018+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:25.025 INFO:teuthology.orchestra.run.smithi165.stdout:4825 read 0x2fa6a90 thru 0x2faabb1 (0x4122 bytes) 2024-09-11T05:52:25.025 INFO:teuthology.orchestra.run.smithi165.stdout:4826 punch from 0x5ecd681 to 0x5ed6f52, (0x98d1 bytes) 2024-09-11T05:52:25.025 INFO:teuthology.orchestra.run.smithi165.stdout:4829 read 0x78c4c55 thru 0x78cf258 (0xa604 bytes) 2024-09-11T05:52:25.026 INFO:teuthology.orchestra.run.smithi165.stdout:4830 punch from 0x6bd8ce4 to 0x6bdcd8b, (0x40a7 bytes) 2024-09-11T05:52:25.026 INFO:teuthology.orchestra.run.smithi165.stdout:4836 read 0xb8bf2f7 thru 0xb8cd142 (0xde4c bytes) 2024-09-11T05:52:25.026 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.026 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:25.039 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:52:25.039 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:25.039 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 91, 2024-09-11T05:52:25.039 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "created": "2024-09-11T05:50:35.802404+0000", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "modified": "2024-09-11T05:52:24.521055+0000", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "last_up_change": "2024-09-11T05:50:49.084592+0000", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "last_in_change": "2024-09-11T05:52:19.634352+0000", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_num": 5804032, 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_set": [ 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "nodeep-scrub", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "pglog_hardlimit", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "purged_snapdirs", 2024-09-11T05:52:25.040 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_deletes", 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "sortbitwise" 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_version": 5, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_snapshot": "", 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_max": 3, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "max_osd": 8, 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "require_min_compat_client": "luminous", 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "min_compat_client": "jewel", 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "squid", 2024-09-11T05:52:25.041 INFO:teuthology.orchestra.run.smithi165.stdout: "allow_crimson": false, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "pools": [ 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 1, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": ".mgr", 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:50.566112+0000", 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 1, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool", 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:52:25.042 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 1, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 1, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 1, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 1, 2024-09-11T05:52:25.043 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 1, 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "18", 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:52:25.044 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 0, 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 0, 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:52:25.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:52:25.046 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:52:25.047 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "options": { 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_max": 32, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_min": 1 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": {} 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:52:25.048 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 6.9000000953674316, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 6.9000000953674316, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.28999999165534973, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 2, 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "rbd", 2024-09-11T05:52:25.049 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:54.170388+0000", 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:52:25.050 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 8, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 8, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 8, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 8, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 8, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "22", 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:52:25.051 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 2, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 22, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:52:25.052 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:52:25.053 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:52:25.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.619999885559082, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.619999885559082, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.86000001430511475, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2.25, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2.25, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.055 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "pool_client.0", 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:51:04.331649+0000", 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:52:25.056 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 32, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 32, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 32, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 32, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 32, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:52:25.057 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "89", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 32, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 89, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:52:25.058 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:52:25.059 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:52:25.060 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.190000057220459, 2024-09-11T05:52:25.061 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.190000057220459, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.86000001430511475, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.875, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 1, 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "osds": [ 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.062 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fdc5357-7501-4582-b301-f5eede8a7bc0", 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 88, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.063 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6815", 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6817", 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6818", 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.065 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6819", 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6822", 2024-09-11T05:52:25.066 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6823", 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6820", 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6821", 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.068 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6817/4049823921", 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6819/4049823921", 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6823/4049823921", 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6821/4049823921", 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:52:25.069 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "78f30980-40a1-46fd-bc1a-a656b32210a9", 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 89, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.070 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6800", 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6801", 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.071 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6802", 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.072 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6803", 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6806", 2024-09-11T05:52:25.073 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6807", 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.074 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6804", 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6805", 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6801/2461301418", 2024-09-11T05:52:25.075 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6803/2461301418", 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6807/2461301418", 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6805/2461301418", 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fab9ebc-408e-4bc1-b3ad-385d0e396f6b", 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.076 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 89, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6808", 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6809", 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6810", 2024-09-11T05:52:25.078 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6811", 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.079 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6814", 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6816", 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6812", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6813", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6809/3166670912", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6811/3166670912", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6816/3166670912", 2024-09-11T05:52:25.081 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6813/3166670912", 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "d3115ede-8d17-4a27-8629-40b19e3c7ee6", 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 90, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6824", 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.083 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6825", 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6826", 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6827", 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6830", 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6831", 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6828", 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6829", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6825/3322140846", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6827/3322140846", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6831/3322140846", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6829/3322140846", 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.087 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "7fa88502-3ffd-4aac-924b-0d6c65737ac6", 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.088 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 90, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6800", 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6801", 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6802", 2024-09-11T05:52:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6803", 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6806", 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6807", 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6804", 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6805", 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:52:25.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6801/347599006", 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6803/347599006", 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6807/347599006", 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6805/347599006", 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "03238cca-4d2b-412d-9fcd-f489cb4a8455", 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 0, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 0, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.095 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6816", 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.096 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6817", 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6818", 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6819", 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6822", 2024-09-11T05:52:25.098 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6823", 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.099 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6820", 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6821", 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6817/4259095655", 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6819/4259095655", 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6823/4259095655", 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6821/4259095655", 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "6c46c2d3-7dd1-4b3a-ad76-391c6e624b40", 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:52:25.102 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6824", 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.103 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6825", 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6826", 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6827", 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6830", 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6831", 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6828", 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6829", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6825/2366978310", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6827/2366978310", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6831/2366978310", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6829/2366978310", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.107 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "f849014a-7b82-4361-8a23-3b7a0be5f5c0", 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:52:25.108 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 89, 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6808", 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6809", 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6810", 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6811", 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6814", 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6815", 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.111 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6812", 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6813", 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.112 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6809/2576837456", 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6811/2576837456", 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6815/2576837456", 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6813/2576837456", 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_xinfo": [ 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.113 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 65536, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.605278+0000", 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.114 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.681171+0000", 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.115 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.531762+0000", 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:46.158901+0000", 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.412535+0000", 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.117 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 65536, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.537269+0000", 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.118 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.433446+0000", 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.612887+0000", 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:52:25.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap": [], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_items": [], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_primaries": [], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_temp": [], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_temp": [], 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklist": {}, 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "range_blocklist": {}, 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profiles": { 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "default": { 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "crush-failure-domain": "osd", 2024-09-11T05:52:25.120 INFO:teuthology.orchestra.run.smithi165.stdout: "k": "2", 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "m": "1", 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin": "jerasure", 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "technique": "reed_sol_van" 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps_queue": [], 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "new_removed_snaps": [], 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "new_purged_snaps": [], 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_node_flags": {}, 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "device_class_flags": {}, 2024-09-11T05:52:25.121 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode": { 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_enabled": false, 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_bucket_count": 0, 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: "degraded_stretch_mode": 0, 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: "recovering_stretch_mode": 0, 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_bucket": 0 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:25.122 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:25.123 INFO:teuthology.orchestra.run.smithi165.stdout:4839 write 0xe75c3eb thru 0xe761242 (0x4e58 bytes) 2024-09-11T05:52:25.124 INFO:teuthology.orchestra.run.smithi165.stdout:4840 read 0x713f61e thru 0x714af8d (0xb970 bytes) 2024-09-11T05:52:25.124 INFO:teuthology.orchestra.run.smithi165.stdout:4841 writesame 0xcd339c4 thru 0xcd37f67 data_size 0x2(0x45a4 bytes) 2024-09-11T05:52:25.124 INFO:teuthology.orchestra.run.smithi165.stdout:4842 writesame 0x9cb2751 thru 0x9cbe22f data_size 0x1(0xbadf bytes) 2024-09-11T05:52:25.125 INFO:tasks.thrashosds.thrasher:j is {'epoch': 91, 'fsid': 'd5628057-c743-43f5-a51d-1915a16ccfdd', 'created': '2024-09-11T05:50:35.802404+0000', 'modified': '2024-09-11T05:52:24.521055+0000', 'last_up_change': '2024-09-11T05:50:49.084592+0000', 'last_in_change': '2024-09-11T05:52:19.634352+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:50:50.566112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:50:54.170388+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 22, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-11T05:51:04.331649+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2fdc5357-7501-4582-b301-f5eede8a7bc0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6815', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 4049823921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 4049823921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 4049823921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 4049823921}]}, 'public_addr': '172.21.15.165:6817/4049823921', 'cluster_addr': '172.21.15.165:6819/4049823921', 'heartbeat_back_addr': '172.21.15.165:6823/4049823921', 'heartbeat_front_addr': '172.21.15.165:6821/4049823921', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '78f30980-40a1-46fd-bc1a-a656b32210a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6800', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2461301418}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2461301418}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 2461301418}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 2461301418}]}, 'public_addr': '172.21.15.165:6801/2461301418', 'cluster_addr': '172.21.15.165:6803/2461301418', 'heartbeat_back_addr': '172.21.15.165:6807/2461301418', 'heartbeat_front_addr': '172.21.15.165:6805/2461301418', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3166670912}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3166670912}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6816', 'nonce': 3166670912}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 3166670912}]}, 'public_addr': '172.21.15.165:6809/3166670912', 'cluster_addr': '172.21.15.165:6811/3166670912', 'heartbeat_back_addr': '172.21.15.165:6816/3166670912', 'heartbeat_front_addr': '172.21.15.165:6813/3166670912', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd3115ede-8d17-4a27-8629-40b19e3c7ee6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 3322140846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 3322140846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 3322140846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 3322140846}]}, 'public_addr': '172.21.15.165:6825/3322140846', 'cluster_addr': '172.21.15.165:6827/3322140846', 'heartbeat_back_addr': '172.21.15.165:6831/3322140846', 'heartbeat_front_addr': '172.21.15.165:6829/3322140846', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 347599006}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 347599006}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 347599006}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 347599006}]}, 'public_addr': '172.21.15.172:6801/347599006', 'cluster_addr': '172.21.15.172:6803/347599006', 'heartbeat_back_addr': '172.21.15.172:6807/347599006', 'heartbeat_front_addr': '172.21.15.172:6805/347599006', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03238cca-4d2b-412d-9fcd-f489cb4a8455', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 4259095655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 4259095655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 4259095655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 4259095655}]}, 'public_addr': '172.21.15.172:6817/4259095655', 'cluster_addr': '172.21.15.172:6819/4259095655', 'heartbeat_back_addr': '172.21.15.172:6823/4259095655', 'heartbeat_front_addr': '172.21.15.172:6821/4259095655', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 2366978310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 2366978310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 2366978310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 2366978310}]}, 'public_addr': '172.21.15.172:6825/2366978310', 'cluster_addr': '172.21.15.172:6827/2366978310', 'heartbeat_back_addr': '172.21.15.172:6831/2366978310', 'heartbeat_front_addr': '172.21.15.172:6829/2366978310', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f849014a-7b82-4361-8a23-3b7a0be5f5c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 89, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 2576837456}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2576837456}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2576837456}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 2576837456}]}, 'public_addr': '172.21.15.172:6809/2576837456', 'cluster_addr': '172.21.15.172:6811/2576837456', 'heartbeat_back_addr': '172.21.15.172:6815/2576837456', 'heartbeat_front_addr': '172.21.15.172:6813/2576837456', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.605278+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.681171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.531762+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:46.158901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.412535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.537269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.433446+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.612887+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T05:52:25.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:4843 read 0x7a1b3fc thru 0x7a2b27a (0xfe7f bytes) 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:4844 write 0xa5044d3 thru 0xa50bac1 (0x75ef bytes) 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:4845 read 0xa2e4c71 thru 0xa2e757e (0x290e bytes) 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:4847 punch from 0xde95205 to 0xdea494d, (0xf748 bytes) 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:4849 read 0x5f8641 thru 0x5f95ad (0xf6d bytes) 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.127 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.128 INFO:teuthology.orchestra.run.smithi165.stdout:4852 trunc from 0xe761243 to 0xde1b806 2024-09-11T05:52:25.128 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:25.120+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:25.136 INFO:teuthology.orchestra.run.smithi165.stdout:4853 read 0x96c08a4 thru 0x96ceb8f (0xe2ec bytes) 2024-09-11T05:52:25.136 INFO:teuthology.orchestra.run.smithi165.stdout:4854 read 0x62cc3f3 thru 0x62d5632 (0x9240 bytes) 2024-09-11T05:52:25.136 INFO:teuthology.orchestra.run.smithi165.stdout:4855 read 0x7a44515 thru 0x7a5448f (0xff7b bytes) 2024-09-11T05:52:25.136 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.136 INFO:teuthology.orchestra.run.smithi165.stdout:4858 writesame 0x6e2a731 thru 0x6e2bf6c data_size 0x2f(0x183c bytes) 2024-09-11T05:52:25.141 INFO:teuthology.orchestra.run.smithi165.stdout:4859 punch from 0x2541780 to 0x254236a, (0xbea bytes) 2024-09-11T05:52:25.141 INFO:teuthology.orchestra.run.smithi165.stdout:4860 read 0xd9c4a41 thru 0xd9cde13 (0x93d3 bytes) 2024-09-11T05:52:25.141 INFO:teuthology.orchestra.run.smithi165.stdout:4861 read 0x7805569 thru 0x78120e6 (0xcb7e bytes) 2024-09-11T05:52:25.142 INFO:teuthology.orchestra.run.smithi165.stdout:4862 write 0xb790067 thru 0xb790216 (0x1b0 bytes) 2024-09-11T05:52:25.144 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.145 INFO:teuthology.orchestra.run.smithi165.stdout:4864 read 0x1514866 thru 0x151a236 (0x59d1 bytes) 2024-09-11T05:52:25.145 INFO:teuthology.orchestra.run.smithi165.stdout:4867 read 0xd303383 thru 0xd3041f0 (0xe6e bytes) 2024-09-11T05:52:25.146 INFO:teuthology.orchestra.run.smithi165.stdout:4871 punch from 0x4bcba20 to 0x4bcde2d, (0x240d bytes) 2024-09-11T05:52:25.146 INFO:teuthology.orchestra.run.smithi165.stdout:4872 read 0xcf069d thru 0xcf526c (0x4bd0 bytes) 2024-09-11T05:52:25.147 INFO:teuthology.orchestra.run.smithi165.stdout:4873 read 0xcbc9319 thru 0xcbd2c56 (0x993e bytes) 2024-09-11T05:52:25.147 INFO:teuthology.orchestra.run.smithi165.stdout:4874 punch from 0xc83822d to 0xc83cadb, (0x48ae bytes) 2024-09-11T05:52:25.147 INFO:teuthology.orchestra.run.smithi165.stdout:4877 punch from 0x87d6083 to 0x87d66a2, (0x61f bytes) 2024-09-11T05:52:25.155 INFO:teuthology.orchestra.run.smithi165.stdout:4878 write 0xe7be3b4 thru 0xe7cd0f7 (0xed44 bytes) 2024-09-11T05:52:25.162 INFO:teuthology.orchestra.run.smithi165.stdout:4879 punch from 0x6b996d6 to 0x6ba5477, (0xbda1 bytes) 2024-09-11T05:52:25.162 INFO:teuthology.orchestra.run.smithi165.stdout:4880 read 0xbcdacc4 thru 0xbcea4d4 (0xf811 bytes) 2024-09-11T05:52:25.162 INFO:teuthology.orchestra.run.smithi165.stdout:4882 write 0x7dd5d40 thru 0x7dd8f52 (0x3213 bytes) 2024-09-11T05:52:25.165 INFO:teuthology.orchestra.run.smithi165.stdout:4884 punch from 0x3cf9c7d to 0x3d0469c, (0xaa1f bytes) 2024-09-11T05:52:25.166 INFO:teuthology.orchestra.run.smithi165.stdout:4885 punch from 0xd87bb22 to 0xd881b77, (0x6055 bytes) 2024-09-11T05:52:25.166 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.166 INFO:teuthology.orchestra.run.smithi165.stdout:4887 write 0x6302f5c thru 0x6308531 (0x55d6 bytes) 2024-09-11T05:52:25.167 INFO:teuthology.orchestra.run.smithi165.stdout:4888 writesame 0x75d3c18 thru 0x75d512f data_size 0x9(0x1518 bytes) 2024-09-11T05:52:25.170 INFO:teuthology.orchestra.run.smithi165.stdout:4889 read 0x8223f34 thru 0x82271ec (0x32b9 bytes) 2024-09-11T05:52:25.170 INFO:teuthology.orchestra.run.smithi165.stdout:4890 punch from 0xc8608c4 to 0xc863f6e, (0x36aa bytes) 2024-09-11T05:52:25.170 INFO:teuthology.orchestra.run.smithi165.stdout:4892 read 0x774bf5c thru 0x7755c73 (0x9d18 bytes) 2024-09-11T05:52:25.171 INFO:teuthology.orchestra.run.smithi165.stdout:4893 read 0xb2e40b5 thru 0xb2f1429 (0xd375 bytes) 2024-09-11T05:52:25.171 INFO:teuthology.orchestra.run.smithi165.stdout:4894 punch from 0x5ca0aa9 to 0x5caf15f, (0xe6b6 bytes) 2024-09-11T05:52:25.181 INFO:teuthology.orchestra.run.smithi165.stdout:4895 trunc from 0xe7cd0f8 to 0x891896f 2024-09-11T05:52:25.194 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.195 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:25.195 INFO:teuthology.orchestra.run.smithi165.stdout:4899 writesame 0x5d5a485 thru 0x5d5a849 data_size 0x1(0x3c5 bytes) 2024-09-11T05:52:25.215 INFO:teuthology.orchestra.run.smithi165.stdout:4900 writesame 0xe61ebf6 thru 0xe62e328 data_size 0xb(0xf733 bytes) 2024-09-11T05:52:25.215 INFO:teuthology.orchestra.run.smithi165.stdout:4901 clone 31 order 20 su 262144 sc 13 2024-09-11T05:52:25.224 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:25.224+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:25.253 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.253 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:25.325 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:25.324+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:25.388 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.388 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:25.425 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:25.424+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:25.433 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:52:25.434 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:52:25.451 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-11T05:52:24.524923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":888041451,"num_objects":1471,"num_object_clones":486,"num_object_copies":2942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":530,"num_read":159588,"num_read_kb":454226,"num_write":16199,"num_write_kb":251616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":67624810,"num_keys_recovered":221,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12135,"ondisk_log_size":12135,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":439264,"kb_used_data":211868,"kb_used_omap":9,"kb_used_meta":227254,"kb_avail":565791776,"statfs":{"total":579820584960,"available":579370778624,"internally_reserved":0,"allocated":216952832,"data_stored":276631965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":232708774},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20030936,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":89,"num_read_kb":138,"num_write":60,"num_write_kb":774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":18163632,"num_keys_recovered":71,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011027"},"pg_stats":[{"pgid":"3.1f","version":"85'328","reported_seq":2012,"reported_epoch":86,"state":"unknown","last_fresh":"2024-09-11T05:52:19.648761+0000","last_change":"2024-09-11T05:52:19.880720+0000","last_active":"2024-09-11T05:52:19.648761+0000","last_peered":"2024-09-11T05:52:19.515917+0000","last_clean":"2024-09-11T05:52:19.515917+0000","last_became_active":"2024-09-11T05:52:17.636132+0000","last_became_peered":"2024-09-11T05:52:17.636132+0000","last_unstale":"2024-09-11T05:52:19.648761+0000","last_undegraded":"2024-09-11T05:52:19.648761+0000","last_fullsized":"2024-09-11T05:52:19.648761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":24763541,"num_objects":57,"num_object_clones":23,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":20,"num_read":1853,"num_read_kb":6240,"num_write":425,"num_write_kb":5071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":2569129,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"88'277","reported_seq":1986,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.528168+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:52:23.528168+0000","last_peered":"2024-09-11T05:52:23.528168+0000","last_clean":"2024-09-11T05:52:23.528168+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:52:23.528168+0000","last_undegraded":"2024-09-11T05:52:23.528168+0000","last_fullsized":"2024-09-11T05:52:23.528168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:38.389931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":27485645,"num_objects":42,"num_object_clones":11,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":17,"num_read":2999,"num_read_kb":7567,"num_write":319,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"85'174","reported_seq":3172,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.643812+0000","last_change":"2024-09-11T05:52:17.641069+0000","last_active":"2024-09-11T05:52:19.643812+0000","last_peered":"2024-09-11T05:52:19.643812+0000","last_clean":"2024-09-11T05:52:19.643812+0000","last_became_active":"2024-09-11T05:52:17.638753+0000","last_became_peered":"2024-09-11T05:52:17.638753+0000","last_unstale":"2024-09-11T05:52:19.643812+0000","last_undegraded":"2024-09-11T05:52:19.643812+0000","last_fullsized":"2024-09-11T05:52:19.643812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:31:14.300120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":25764660,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":13,"num_read":2726,"num_read_kb":13249,"num_write":288,"num_write_kb":6083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"85'347","reported_seq":3067,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.643683+0000","last_change":"2024-09-11T05:52:17.640186+0000","last_active":"2024-09-11T05:52:19.643683+0000","last_peered":"2024-09-11T05:52:19.643683+0000","last_clean":"2024-09-11T05:52:19.643683+0000","last_became_active":"2024-09-11T05:52:17.637949+0000","last_became_peered":"2024-09-11T05:52:17.637949+0000","last_unstale":"2024-09-11T05:52:19.643683+0000","last_undegraded":"2024-09-11T05:52:19.643683+0000","last_fullsized":"2024-09-11T05:52:19.643683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:17.226064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":7706676,"num_objects":38,"num_object_clones":14,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":12,"num_read":3619,"num_read_kb":7504,"num_write":466,"num_write_kb":3568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"85'391","reported_seq":6143,"reported_epoch":86,"state":"unknown","last_fresh":"2024-09-11T05:52:19.644789+0000","last_change":"2024-09-11T05:52:19.881596+0000","last_active":"2024-09-11T05:52:19.644789+0000","last_peered":"2024-09-11T05:52:18.767378+0000","last_clean":"2024-09-11T05:52:18.767378+0000","last_became_active":"2024-09-11T05:51:05.512501+0000","last_became_peered":"2024-09-11T05:51:05.512501+0000","last_unstale":"2024-09-11T05:52:19.644789+0000","last_undegraded":"2024-09-11T05:52:19.644789+0000","last_fullsized":"2024-09-11T05:52:19.644789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":21585641,"num_objects":40,"num_object_clones":13,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":13,"num_read":6837,"num_read_kb":21142,"num_write":515,"num_write_kb":8738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"85'266","reported_seq":7311,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.643564+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:52:19.643564+0000","last_peered":"2024-09-11T05:52:19.643564+0000","last_clean":"2024-09-11T05:52:19.643564+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:52:19.643564+0000","last_undegraded":"2024-09-11T05:52:19.643564+0000","last_fullsized":"2024-09-11T05:52:19.643564+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:42.094931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":27789460,"num_objects":50,"num_object_clones":16,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":22,"num_read":7075,"num_read_kb":12822,"num_write":369,"num_write_kb":11903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"85'520","reported_seq":3613,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.639618+0000","last_change":"2024-09-11T05:52:08.536765+0000","last_active":"2024-09-11T05:52:19.639618+0000","last_peered":"2024-09-11T05:52:19.639618+0000","last_clean":"2024-09-11T05:52:19.639618+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:52:19.639618+0000","last_undegraded":"2024-09-11T05:52:19.639618+0000","last_fullsized":"2024-09-11T05:52:19.639618+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:41:20.336500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":14837078,"num_objects":57,"num_object_clones":17,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":21,"num_read":4614,"num_read_kb":9503,"num_write":747,"num_write_kb":5751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"88'323","reported_seq":6816,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.532280+0000","last_change":"2024-09-11T05:52:23.532280+0000","last_active":"2024-09-11T05:52:23.532280+0000","last_peered":"2024-09-11T05:52:23.532280+0000","last_clean":"2024-09-11T05:52:23.532280+0000","last_became_active":"2024-09-11T05:52:23.529775+0000","last_became_peered":"2024-09-11T05:52:23.529775+0000","last_unstale":"2024-09-11T05:52:23.532280+0000","last_undegraded":"2024-09-11T05:52:23.532280+0000","last_fullsized":"2024-09-11T05:52:23.532280+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:09:38.716786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":32893402,"num_objects":54,"num_object_clones":19,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":15,"num_read":7518,"num_read_kb":25992,"num_write":384,"num_write_kb":9926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":3756409,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"89'274","reported_seq":2634,"reported_epoch":90,"state":"peering","last_fresh":"2024-09-11T05:52:23.528874+0000","last_change":"2024-09-11T05:52:23.528874+0000","last_active":"2024-09-11T05:52:23.528874+0000","last_peered":"2024-09-11T05:52:22.569953+0000","last_clean":"2024-09-11T05:52:19.587015+0000","last_became_active":"2024-09-11T05:52:21.574125+0000","last_became_peered":"2024-09-11T05:52:21.574125+0000","last_unstale":"2024-09-11T05:52:23.528874+0000","last_undegraded":"2024-09-11T05:52:23.528874+0000","last_fullsized":"2024-09-11T05:52:23.528874+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":9660057,"num_objects":47,"num_object_clones":16,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":18,"num_read":2713,"num_read_kb":5879,"num_write":368,"num_write_kb":5246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":2921918,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"78'195","reported_seq":8659,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648679+0000","last_change":"2024-09-11T05:52:17.637099+0000","last_active":"2024-09-11T05:52:19.648679+0000","last_peered":"2024-09-11T05:52:19.648679+0000","last_clean":"2024-09-11T05:52:19.648679+0000","last_became_active":"2024-09-11T05:52:17.635558+0000","last_became_peered":"2024-09-11T05:52:17.635558+0000","last_unstale":"2024-09-11T05:52:19.648679+0000","last_undegraded":"2024-09-11T05:52:19.648679+0000","last_fullsized":"2024-09-11T05:52:19.648679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:39:21.174731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":34014943,"num_objects":40,"num_object_clones":12,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":19,"num_read":8189,"num_read_kb":22906,"num_write":264,"num_write_kb":12235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"85'327","reported_seq":7032,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648429+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:52:19.648429+0000","last_peered":"2024-09-11T05:52:19.648429+0000","last_clean":"2024-09-11T05:52:19.648429+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:52:19.648429+0000","last_undegraded":"2024-09-11T05:52:19.648429+0000","last_fullsized":"2024-09-11T05:52:19.648429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:44:24.172633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":29680346,"num_objects":34,"num_object_clones":10,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":12,"num_read":7654,"num_read_kb":9871,"num_write":335,"num_write_kb":7773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"85'658","reported_seq":5028,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.643977+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:52:19.643977+0000","last_peered":"2024-09-11T05:52:19.643977+0000","last_clean":"2024-09-11T05:52:19.643977+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:52:19.643977+0000","last_undegraded":"2024-09-11T05:52:19.643977+0000","last_fullsized":"2024-09-11T05:52:19.643977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:13.630967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":51219433,"num_objects":63,"num_object_clones":25,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":19,"num_read":6759,"num_read_kb":20585,"num_write":908,"num_write_kb":12458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.649039+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:52:19.649039+0000","last_peered":"2024-09-11T05:52:19.649039+0000","last_clean":"2024-09-11T05:52:19.649039+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:52:19.649039+0000","last_undegraded":"2024-09-11T05:52:19.649039+0000","last_fullsized":"2024-09-11T05:52:19.649039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"85'558","reported_seq":4088,"reported_epoch":86,"state":"unknown","last_fresh":"2024-09-11T05:52:19.640894+0000","last_change":"2024-09-11T05:52:19.879196+0000","last_active":"2024-09-11T05:52:19.640894+0000","last_peered":"2024-09-11T05:52:19.519832+0000","last_clean":"2024-09-11T05:52:19.519832+0000","last_became_active":"2024-09-11T05:52:08.529499+0000","last_became_peered":"2024-09-11T05:52:08.529499+0000","last_unstale":"2024-09-11T05:52:19.640894+0000","last_undegraded":"2024-09-11T05:52:19.640894+0000","last_fullsized":"2024-09-11T05:52:19.640894+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":24601473,"num_objects":53,"num_object_clones":18,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":21,"num_read":4753,"num_read_kb":11109,"num_write":786,"num_write_kb":5841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4454693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.640089+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:52:19.640089+0000","last_peered":"2024-09-11T05:52:19.640089+0000","last_clean":"2024-09-11T05:52:19.640089+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:52:19.640089+0000","last_undegraded":"2024-09-11T05:52:19.640089+0000","last_fullsized":"2024-09-11T05:52:19.640089+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"77'342","reported_seq":2127,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.528285+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:52:23.528285+0000","last_peered":"2024-09-11T05:52:23.528285+0000","last_clean":"2024-09-11T05:52:23.528285+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:52:23.528285+0000","last_undegraded":"2024-09-11T05:52:23.528285+0000","last_fullsized":"2024-09-11T05:52:23.528285+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:00:17.257717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":10308920,"num_objects":34,"num_object_clones":11,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":15,"num_read":2605,"num_read_kb":8054,"num_write":522,"num_write_kb":4281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":145,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T05:52:18.697730+0000","last_change":"2024-09-11T05:52:19.650265+0000","last_active":"2024-09-11T05:52:19.650265+0000","last_peered":"2024-09-11T05:52:18.697730+0000","last_clean":"2024-09-11T05:52:18.697730+0000","last_became_active":"2024-09-11T05:50:56.481080+0000","last_became_peered":"2024-09-11T05:50:56.481080+0000","last_unstale":"2024-09-11T05:52:18.697730+0000","last_undegraded":"2024-09-11T05:52:18.697730+0000","last_fullsized":"2024-09-11T05:52:18.697730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'229","reported_seq":4874,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.639634+0000","last_change":"2024-09-11T05:52:08.530178+0000","last_active":"2024-09-11T05:52:19.639634+0000","last_peered":"2024-09-11T05:52:19.639634+0000","last_clean":"2024-09-11T05:52:19.639634+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:52:19.639634+0000","last_undegraded":"2024-09-11T05:52:19.639634+0000","last_fullsized":"2024-09-11T05:52:19.639634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:28:55.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":22267576,"num_objects":26,"num_object_clones":10,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":8,"num_read":4441,"num_read_kb":22308,"num_write":245,"num_write_kb":5923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T05:52:18.632183+0000","last_change":"2024-09-11T05:52:19.639984+0000","last_active":"2024-09-11T05:52:19.639984+0000","last_peered":"2024-09-11T05:52:18.632183+0000","last_clean":"2024-09-11T05:52:18.632183+0000","last_became_active":"2024-09-11T05:52:05.512365+0000","last_became_peered":"2024-09-11T05:52:05.512365+0000","last_unstale":"2024-09-11T05:52:18.632183+0000","last_undegraded":"2024-09-11T05:52:18.632183+0000","last_fullsized":"2024-09-11T05:52:18.632183+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"90'378","reported_seq":4766,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.720700+0000","last_change":"2024-09-11T05:52:18.643055+0000","last_active":"2024-09-11T05:52:23.720700+0000","last_peered":"2024-09-11T05:52:23.720700+0000","last_clean":"2024-09-11T05:52:23.720700+0000","last_became_active":"2024-09-11T05:52:18.641130+0000","last_became_peered":"2024-09-11T05:52:18.641130+0000","last_unstale":"2024-09-11T05:52:23.720700+0000","last_undegraded":"2024-09-11T05:52:23.720700+0000","last_fullsized":"2024-09-11T05:52:23.720700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:34:45.618566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":52034601,"num_objects":46,"num_object_clones":15,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":16,"num_read":6112,"num_read_kb":18947,"num_write":415,"num_write_kb":8551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"82'471","reported_seq":2680,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T05:52:18.983266+0000","last_change":"2024-09-11T05:52:19.645452+0000","last_active":"2024-09-11T05:52:19.645452+0000","last_peered":"2024-09-11T05:52:18.983266+0000","last_clean":"2024-09-11T05:52:18.983266+0000","last_became_active":"2024-09-11T05:51:05.516271+0000","last_became_peered":"2024-09-11T05:51:05.516271+0000","last_unstale":"2024-09-11T05:52:18.983266+0000","last_undegraded":"2024-09-11T05:52:18.983266+0000","last_fullsized":"2024-09-11T05:52:18.983266+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":9886265,"num_objects":41,"num_object_clones":13,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":15,"num_read":3836,"num_read_kb":8249,"num_write":700,"num_write_kb":4034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.639562+0000","last_change":"2024-09-11T05:52:14.615928+0000","last_active":"2024-09-11T05:52:19.639562+0000","last_peered":"2024-09-11T05:52:19.639562+0000","last_clean":"2024-09-11T05:52:19.639562+0000","last_became_active":"2024-09-11T05:52:14.609394+0000","last_became_peered":"2024-09-11T05:52:14.609394+0000","last_unstale":"2024-09-11T05:52:19.639562+0000","last_undegraded":"2024-09-11T05:52:19.639562+0000","last_fullsized":"2024-09-11T05:52:19.639562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:25.789248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"83'460","reported_seq":2580,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.639752+0000","last_change":"2024-09-11T05:52:08.529630+0000","last_active":"2024-09-11T05:52:19.639752+0000","last_peered":"2024-09-11T05:52:19.639752+0000","last_clean":"2024-09-11T05:52:19.639752+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:52:19.639752+0000","last_undegraded":"2024-09-11T05:52:19.639752+0000","last_fullsized":"2024-09-11T05:52:19.639752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:24.716942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":18211446,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":17,"num_read":3812,"num_read_kb":9361,"num_write":538,"num_write_kb":4956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648602+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:52:19.648602+0000","last_peered":"2024-09-11T05:52:19.648602+0000","last_clean":"2024-09-11T05:52:19.648602+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:52:19.648602+0000","last_undegraded":"2024-09-11T05:52:19.648602+0000","last_fullsized":"2024-09-11T05:52:19.648602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"79'479","reported_seq":4607,"reported_epoch":86,"state":"unknown","last_fresh":"2024-09-11T05:52:19.642487+0000","last_change":"2024-09-11T05:52:19.881531+0000","last_active":"2024-09-11T05:52:19.642487+0000","last_peered":"2024-09-11T05:52:11.893952+0000","last_clean":"2024-09-11T05:52:11.893952+0000","last_became_active":"2024-09-11T05:51:05.512790+0000","last_became_peered":"2024-09-11T05:51:05.512790+0000","last_unstale":"2024-09-11T05:52:19.642487+0000","last_undegraded":"2024-09-11T05:52:19.642487+0000","last_fullsized":"2024-09-11T05:52:19.642487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":28942436,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":22,"num_read":7220,"num_read_kb":17134,"num_write":603,"num_write_kb":7517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.647564+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:52:19.647564+0000","last_peered":"2024-09-11T05:52:19.647564+0000","last_clean":"2024-09-11T05:52:19.647564+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:52:19.647564+0000","last_undegraded":"2024-09-11T05:52:19.647564+0000","last_fullsized":"2024-09-11T05:52:19.647564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"88'330","reported_seq":6349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.527651+0000","last_change":"2024-09-11T05:52:17.638710+0000","last_active":"2024-09-11T05:52:23.527651+0000","last_peered":"2024-09-11T05:52:23.527651+0000","last_clean":"2024-09-11T05:52:23.527651+0000","last_became_active":"2024-09-11T05:52:17.637965+0000","last_became_peered":"2024-09-11T05:52:17.637965+0000","last_unstale":"2024-09-11T05:52:23.527651+0000","last_undegraded":"2024-09-11T05:52:23.527651+0000","last_fullsized":"2024-09-11T05:52:23.527651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:28.712612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":40878516,"num_objects":49,"num_object_clones":18,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":17,"num_read":6263,"num_read_kb":17182,"num_write":467,"num_write_kb":12022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648714+0000","last_change":"2024-09-11T05:52:14.618897+0000","last_active":"2024-09-11T05:52:19.648714+0000","last_peered":"2024-09-11T05:52:19.648714+0000","last_clean":"2024-09-11T05:52:19.648714+0000","last_became_active":"2024-09-11T05:52:14.614711+0000","last_became_peered":"2024-09-11T05:52:14.614711+0000","last_unstale":"2024-09-11T05:52:19.648714+0000","last_undegraded":"2024-09-11T05:52:19.648714+0000","last_fullsized":"2024-09-11T05:52:19.648714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:51.273072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":296,"reported_epoch":85,"state":"peering","last_fresh":"2024-09-11T05:52:18.697608+0000","last_change":"2024-09-11T05:52:19.647901+0000","last_active":"2024-09-11T05:52:19.647901+0000","last_peered":"2024-09-11T05:52:18.697608+0000","last_clean":"2024-09-11T05:52:18.697608+0000","last_became_active":"2024-09-11T05:50:52.121434+0000","last_became_peered":"2024-09-11T05:50:52.121434+0000","last_unstale":"2024-09-11T05:52:18.697608+0000","last_undegraded":"2024-09-11T05:52:18.697608+0000","last_fullsized":"2024-09-11T05:52:18.697608+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"90'266","reported_seq":5245,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.528024+0000","last_change":"2024-09-11T05:52:08.528671+0000","last_active":"2024-09-11T05:52:23.528024+0000","last_peered":"2024-09-11T05:52:23.528024+0000","last_clean":"2024-09-11T05:52:23.528024+0000","last_became_active":"2024-09-11T05:52:08.527718+0000","last_became_peered":"2024-09-11T05:52:08.527718+0000","last_unstale":"2024-09-11T05:52:23.528024+0000","last_undegraded":"2024-09-11T05:52:23.528024+0000","last_fullsized":"2024-09-11T05:52:23.528024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:53.612282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":34799819,"num_objects":48,"num_object_clones":17,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":15,"num_read":4665,"num_read_kb":14812,"num_write":317,"num_write_kb":16360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"85'190","reported_seq":2736,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.643463+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:52:19.643463+0000","last_peered":"2024-09-11T05:52:19.643463+0000","last_clean":"2024-09-11T05:52:19.643463+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:52:19.643463+0000","last_undegraded":"2024-09-11T05:52:19.643463+0000","last_fullsized":"2024-09-11T05:52:19.643463+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:51.697135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":8665123,"num_objects":46,"num_object_clones":15,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":19,"num_read":2473,"num_read_kb":10303,"num_write":171,"num_write_kb":2687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"85'433","reported_seq":4498,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.645917+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:52:19.645917+0000","last_peered":"2024-09-11T05:52:19.645917+0000","last_clean":"2024-09-11T05:52:19.645917+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:52:19.645917+0000","last_undegraded":"2024-09-11T05:52:19.645917+0000","last_fullsized":"2024-09-11T05:52:19.645917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:49:38.026580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39628839,"num_objects":45,"num_object_clones":13,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":16,"num_read":5823,"num_read_kb":13358,"num_write":536,"num_write_kb":10895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'271","reported_seq":2522,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.535746+0000","last_change":"2024-09-11T05:52:23.535746+0000","last_active":"2024-09-11T05:52:23.535746+0000","last_peered":"2024-09-11T05:52:23.535746+0000","last_clean":"2024-09-11T05:52:23.535746+0000","last_became_active":"2024-09-11T05:52:23.534658+0000","last_became_peered":"2024-09-11T05:52:23.534658+0000","last_unstale":"2024-09-11T05:52:23.535746+0000","last_undegraded":"2024-09-11T05:52:23.535746+0000","last_fullsized":"2024-09-11T05:52:23.535746+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:40:54.679983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":28250130,"num_objects":47,"num_object_clones":16,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":18,"num_read":2432,"num_read_kb":14776,"num_write":320,"num_write_kb":8396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":3743621,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"88'280","reported_seq":4899,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.528038+0000","last_change":"2024-09-11T05:52:18.644312+0000","last_active":"2024-09-11T05:52:23.528038+0000","last_peered":"2024-09-11T05:52:23.528038+0000","last_clean":"2024-09-11T05:52:23.528038+0000","last_became_active":"2024-09-11T05:52:18.643189+0000","last_became_peered":"2024-09-11T05:52:18.643189+0000","last_unstale":"2024-09-11T05:52:23.528038+0000","last_undegraded":"2024-09-11T05:52:23.528038+0000","last_fullsized":"2024-09-11T05:52:23.528038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:58:26.282298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":12629852,"num_objects":37,"num_object_clones":9,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":13,"num_read":4718,"num_read_kb":7876,"num_write":374,"num_write_kb":7136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"85'309","reported_seq":4207,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.649054+0000","last_change":"2024-09-11T05:52:17.643682+0000","last_active":"2024-09-11T05:52:19.649054+0000","last_peered":"2024-09-11T05:52:19.649054+0000","last_clean":"2024-09-11T05:52:19.649054+0000","last_became_active":"2024-09-11T05:52:17.641884+0000","last_became_peered":"2024-09-11T05:52:17.641884+0000","last_unstale":"2024-09-11T05:52:19.649054+0000","last_undegraded":"2024-09-11T05:52:19.649054+0000","last_fullsized":"2024-09-11T05:52:19.649054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:45:09.081160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":16990521,"num_objects":44,"num_object_clones":15,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":17,"num_read":4274,"num_read_kb":15379,"num_write":407,"num_write_kb":4990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"85'777","reported_seq":7904,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648970+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:52:19.648970+0000","last_peered":"2024-09-11T05:52:19.648970+0000","last_clean":"2024-09-11T05:52:19.648970+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:52:19.648970+0000","last_undegraded":"2024-09-11T05:52:19.648970+0000","last_fullsized":"2024-09-11T05:52:19.648970+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:28.055268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":60622087,"num_objects":48,"num_object_clones":14,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":14,"num_read":9866,"num_read_kb":28009,"num_write":1058,"num_write_kb":14518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"77'356","reported_seq":2499,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648749+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:52:19.648749+0000","last_peered":"2024-09-11T05:52:19.648749+0000","last_clean":"2024-09-11T05:52:19.648749+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:52:19.648749+0000","last_undegraded":"2024-09-11T05:52:19.648749+0000","last_fullsized":"2024-09-11T05:52:19.648749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:32:41.126040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":48968728,"num_objects":45,"num_object_clones":14,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":18,"num_read":3045,"num_read_kb":13551,"num_write":552,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"90'677","reported_seq":2564,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T05:52:23.718342+0000","last_change":"2024-09-11T05:52:23.539648+0000","last_active":"2024-09-11T05:52:23.718342+0000","last_peered":"2024-09-11T05:52:23.718342+0000","last_clean":"2024-09-11T05:52:23.718342+0000","last_became_active":"2024-09-11T05:52:23.536753+0000","last_became_peered":"2024-09-11T05:52:23.536753+0000","last_unstale":"2024-09-11T05:52:23.718342+0000","last_undegraded":"2024-09-11T05:52:23.718342+0000","last_fullsized":"2024-09-11T05:52:23.718342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:02:39.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43945741,"num_objects":51,"num_object_clones":18,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":18,"num_read":4545,"num_read_kb":20758,"num_write":1091,"num_write_kb":10077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":7741684,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"85'352","reported_seq":3791,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.647638+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:52:19.647638+0000","last_peered":"2024-09-11T05:52:19.647638+0000","last_clean":"2024-09-11T05:52:19.647638+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:52:19.647638+0000","last_undegraded":"2024-09-11T05:52:19.647638+0000","last_fullsized":"2024-09-11T05:52:19.647638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:05.855338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":23309720,"num_objects":55,"num_object_clones":21,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":19,"num_read":4475,"num_read_kb":10750,"num_write":473,"num_write_kb":4901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"85'522","reported_seq":2786,"reported_epoch":86,"state":"unknown","last_fresh":"2024-09-11T05:52:19.648612+0000","last_change":"2024-09-11T05:52:19.894736+0000","last_active":"2024-09-11T05:52:19.648612+0000","last_peered":"2024-09-11T05:52:19.388477+0000","last_clean":"2024-09-11T05:52:19.388477+0000","last_became_active":"2024-09-11T05:51:05.502205+0000","last_became_peered":"2024-09-11T05:51:05.502205+0000","last_unstale":"2024-09-11T05:52:19.648612+0000","last_undegraded":"2024-09-11T05:52:19.648612+0000","last_fullsized":"2024-09-11T05:52:19.648612+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":9355868,"num_objects":38,"num_object_clones":13,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":15,"num_read":4371,"num_read_kb":12500,"num_write":730,"num_write_kb":4558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"78'255","reported_seq":7064,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-11T05:52:19.648062+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:52:19.648062+0000","last_peered":"2024-09-11T05:52:19.648062+0000","last_clean":"2024-09-11T05:52:19.648062+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:52:19.648062+0000","last_undegraded":"2024-09-11T05:52:19.648062+0000","last_fullsized":"2024-09-11T05:52:19.648062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:54.980896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":45752521,"num_objects":51,"num_object_clones":18,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":16,"num_read":7187,"num_read_kb":16450,"num_write":345,"num_write_kb":11475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":887451064,"num_objects":1467,"num_object_clones":486,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":530,"num_read":159472,"num_read_kb":454126,"num_write":16038,"num_write_kb":249198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1684,"num_bytes_recovered":67624791,"num_keys_recovered":221,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273027072,"data_stored":356366591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12015,"ondisk_log_size":12015,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83120,"kb_used_data":38040,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94288720,"statfs":{"total":96636764160,"available":96551649280,"internally_reserved":0,"allocated":38952960,"data_stored":50159941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64184,"kb_used_data":28828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307656,"statfs":{"total":96636764160,"available":96571039744,"internally_reserved":0,"allocated":29519872,"data_stored":37458055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":60129542166,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103624,"kb_used_data":54392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94268216,"statfs":{"total":96636764160,"available":96530653184,"internally_reserved":0,"allocated":55697408,"data_stored":74642115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77636,"kb_used_data":50612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94294204,"statfs":{"total":96636764160,"available":96557264896,"internally_reserved":0,"allocated":51826688,"data_stored":63165365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45876,"kb_used_data":14688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325964,"statfs":{"total":96636764160,"available":96589787136,"internally_reserved":0,"allocated":15040512,"data_stored":19575929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64824,"kb_used_data":25308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94307016,"statfs":{"total":96636764160,"available":96570384384,"internally_reserved":0,"allocated":25915392,"data_stored":31630560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":60129542167,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37187584,"data_stored":47508326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24899584,"data_stored":31167312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":19206314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50786304,"data_stored":62683221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54779904,"data_stored":74258223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25153536,"data_stored":35244226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28692480,"data_stored":37157748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":37392384,"data_stored":49141221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:52:25.453 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [7, 6, 4, 2] 2024-09-11T05:52:25.453 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '7', '6', '4', '2'] 2024-09-11T05:52:25.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.1 7 6 4 2 2024-09-11T05:52:25.519 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.519 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:25.540 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:25.540+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:25.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:52:25.627 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:25.624+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:25.653 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.653 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:25.719 INFO:teuthology.orchestra.run.smithi165.stdout:leaving image image_client.0-clone30 intact 2024-09-11T05:52:25.729 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:25.726+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:25.795 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.795 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:25.854 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:25.854+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:25.930 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-09-11T05:52:25.931 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:25.930+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:25.977 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:25.977 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:25.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:26.031 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:26.028+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:26.133 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:26.130+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:26.152 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:26.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:26.234 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:26.232+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:26.327 INFO:teuthology.orchestra.run.smithi165.stdout:4903 read 0x2ba8338 thru 0x2bb38b6 (0xb57f bytes) 2024-09-11T05:52:26.328 INFO:teuthology.orchestra.run.smithi165.stdout:4904 read 0x8fb726d thru 0x8fc16e0 (0xa474 bytes) 2024-09-11T05:52:26.329 INFO:teuthology.orchestra.run.smithi165.stdout:4905 write 0x9b5ea66 thru 0x9b6dc4f (0xf1ea bytes) 2024-09-11T05:52:26.335 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:26.334+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:26.336 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:26.336 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:26.346 INFO:teuthology.orchestra.run.smithi165.stdout:4906 read 0xc0d518c thru 0xc0d9abb (0x4930 bytes) 2024-09-11T05:52:26.346 INFO:teuthology.orchestra.run.smithi165.stdout:4908 read 0x3ca4ec1 thru 0x3cb3230 (0xe370 bytes) 2024-09-11T05:52:26.346 INFO:teuthology.orchestra.run.smithi165.stdout:4910 read 0xa036091 thru 0xa038dfe (0x2d6e bytes) 2024-09-11T05:52:26.346 INFO:teuthology.orchestra.run.smithi165.stdout:4911 writesame 0x6d6691d thru 0x6d7322b data_size 0x3f(0xc90f bytes) 2024-09-11T05:52:26.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:26.356 INFO:teuthology.orchestra.run.smithi165.stdout:4912 read 0x9108e1a thru 0x9115ac1 (0xcca8 bytes) 2024-09-11T05:52:26.356 INFO:teuthology.orchestra.run.smithi165.stdout:4914 read 0x4285aa3 thru 0x4293f5e (0xe4bc bytes) 2024-09-11T05:52:26.357 INFO:teuthology.orchestra.run.smithi165.stdout:4915 read 0x9946549 thru 0x994df55 (0x7a0d bytes) 2024-09-11T05:52:26.371 INFO:teuthology.orchestra.run.smithi165.stdout:4918 trunc from 0xe62e329 to 0x2fa7da7 2024-09-11T05:52:26.418 INFO:teuthology.orchestra.run.smithi165.stdout:4919 read 0x2ccaa4 thru 0x2d1dda (0x5337 bytes) 2024-09-11T05:52:26.433 INFO:teuthology.orchestra.run.smithi165.stdout:4920 write 0x811cbb1 thru 0x812354e (0x699e bytes) 2024-09-11T05:52:26.435 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:26.434+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:26.440 INFO:teuthology.orchestra.run.smithi165.stdout:4921 read 0x7a1cc07 thru 0x7a27e11 (0xb20b bytes) 2024-09-11T05:52:26.459 INFO:teuthology.orchestra.run.smithi165.stdout:4922 write 0xe9d38e7 thru 0xe9de306 (0xaa20 bytes) 2024-09-11T05:52:26.465 INFO:teuthology.orchestra.run.smithi165.stdout:4923 read 0xbe7c37e thru 0xbe83e56 (0x7ad9 bytes) 2024-09-11T05:52:26.466 INFO:teuthology.orchestra.run.smithi165.stdout:4924 writesame 0xb1f8a06 thru 0xb206daf data_size 0x2e(0xe3aa bytes) 2024-09-11T05:52:26.476 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:26.476 INFO:teuthology.orchestra.run.smithi165.stdout:4926 write 0xc16fdb4 thru 0xc17a472 (0xa6bf bytes) 2024-09-11T05:52:26.479 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:26.479 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:26.482 INFO:teuthology.orchestra.run.smithi165.stdout:4927 clone 32 order 21 su 32768 sc 13 2024-09-11T05:52:26.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:26.536 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:26.534+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:26.637 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:26.636+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:26.651 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:26.651 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:26.662 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:52:26.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:26.668 INFO:teuthology.orchestra.run.smithi165.stderr:set 2.1 pg_upmap_items mapping to [7->6,4->2] 2024-09-11T05:52:26.738 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:26.738+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:26.773 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone31 from 0xe9de307 (overlap 0x2fa7da7) to 0x0 2024-09-11T05:52:26.799 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:26.799 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:26.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:26.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:26.839 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:26.838+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:26.940 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:26.938+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:26.953 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:26.953 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:26.954 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-09-11T05:52:26.970 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:26.971 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:26.987 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:26.987 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:26.994 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:26.994 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:26.994 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:26.994 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:27.004 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:27.004 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:27.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:27.025 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:27.025 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:27.041 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:27.040+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:27.047 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:27.047 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:27.068 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:27.069 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:27.089 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:27.089 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:27.137 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:27.137 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:27.137 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:27.137 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:27.138 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:27.142 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:27.139+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:27.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:27.243 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:27.239+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:27.315 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965381002000001, 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965351593000001, 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965331536000001, 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965290148999998, 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965257667000003, 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965237610999999, 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965218108999998, 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965193300000003, 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965174150999999, 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965153336999997, 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965133504000001, 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965113563999999, 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965087443000002, 2024-09-11T05:52:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.965050859000002, 2024-09-11T05:52:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.960263060000003, 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.960188313000003, 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.960115553999998, 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.960094634000001, 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 45.618128443000003, 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.909689031999999, 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.419 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:27.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4928 read 0x5c279d1 thru 0x5c2d73f (0x5d6f bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4929 read 0xd2c578 thru 0xd3ad0b (0xe794 bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4930 write 0xe6b3818 thru 0xe6b6787 (0x2f70 bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4932 read 0x866f13b thru 0x867c1f7 (0xd0bd bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4933 trunc from 0xe9de307 to 0x440decd 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4935 read 0x1c6fca8 thru 0x1c7561a (0x5973 bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4936 read 0x108accc thru 0x1093623 (0x8958 bytes) 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:27.422 INFO:teuthology.orchestra.run.smithi165.stdout:4939 trunc from 0x440decd to 0x83e06d2 2024-09-11T05:52:27.423 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:27.419+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:27.432 INFO:teuthology.orchestra.run.smithi165.stdout:4940 read 0x8215cda thru 0x82187d5 (0x2afc bytes) 2024-09-11T05:52:27.451 INFO:teuthology.orchestra.run.smithi165.stdout:4941 write 0xe057eb2 thru 0xe06009a (0x81e9 bytes) 2024-09-11T05:52:27.479 INFO:teuthology.orchestra.run.smithi165.stdout:4942 write 0xecc42fd thru 0xecc454c (0x250 bytes) 2024-09-11T05:52:27.484 INFO:teuthology.orchestra.run.smithi165.stdout:4943 read 0x629301e thru 0x62968f6 (0x38d9 bytes) 2024-09-11T05:52:27.490 INFO:teuthology.orchestra.run.smithi165.stdout:4944 trunc from 0xecc454d to 0xa9fb4db 2024-09-11T05:52:27.512 INFO:teuthology.orchestra.run.smithi165.stdout:4945 read 0xe0702c thru 0xe0ef89 (0x7f5e bytes) 2024-09-11T05:52:27.513 INFO:teuthology.orchestra.run.smithi165.stdout:4946 write 0x995b8d5 thru 0x99690d9 (0xd805 bytes) 2024-09-11T05:52:27.520 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:27.519+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:27.523 INFO:teuthology.orchestra.run.smithi165.stdout:4948 write 0x6310d47 thru 0x63132c6 (0x2580 bytes) 2024-09-11T05:52:27.543 INFO:teuthology.orchestra.run.smithi165.stdout:4949 trunc from 0xa9fb4db to 0x180b063 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128143 3.1f 3:ff4b6949:::rbd_data.112438b89d6.000000000000000d:22 [read 327680~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565257+0000", 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0013463080000000001, 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013739379999999999, 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.567 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128143 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565257+0000", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565257+0000", 2024-09-11T05:52:27.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565257+0000", 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565258+0000", 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.203e-06 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565261+0000", 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2959999999999999e-06 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565264+0000", 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0539999999999999e-06 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566536+0000", 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012713049999999999 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566646+0000", 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011040300000000001 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128145 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565310+0000", 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001292896, 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001399899, 2024-09-11T05:52:27.571 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128145 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565310+0000", 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565310+0000", 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565311+0000", 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2300000000000001e-07 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565314+0000", 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.128e-06 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565329+0000", 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4983e-05 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565335+0000", 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2990000000000002e-06 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.574 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128146 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565344+0000", 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001258534, 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014239980000000001, 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.575 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128146 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565344+0000", 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565344+0000", 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565345+0000", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2900000000000004e-07 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565352+0000", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565359+0000", 2024-09-11T05:52:27.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6000000000000003e-06 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565363+0000", 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9309999999999999e-06 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128175 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565516+0000", 2024-09-11T05:52:27.578 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00108734, 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001312459, 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128175 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565516+0000", 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565516+0000", 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565517+0000", 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.19e-06 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565523+0000", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0669999999999997e-06 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565535+0000", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2204e-05 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565544+0000", 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1160000000000005e-06 2024-09-11T05:52:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128184 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565690+0000", 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00091270700000000001, 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001190766, 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128184 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565690+0000", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565690+0000", 2024-09-11T05:52:27.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565691+0000", 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9999999999999995e-07 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565696+0000", 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9579999999999998e-06 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565707+0000", 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0933e-05 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565716+0000", 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4139999999999994e-06 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128185 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565745+0000", 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00085784299999999995, 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001189905, 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128185 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565745+0000", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565745+0000", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565746+0000", 2024-09-11T05:52:27.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2699999999999998e-07 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565758+0000", 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1892000000000001e-05 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565767+0000", 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6169999999999996e-06 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565775+0000", 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0249999999999997e-06 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128187 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.565786+0000", 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00081667600000000001, 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001200336, 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.589 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128187 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565786+0000", 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565786+0000", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565787+0000", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565790+0000", 2024-09-11T05:52:27.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8320000000000002e-06 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565798+0000", 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1750000000000005e-06 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.565805+0000", 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6420000000000001e-06 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.592 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128231 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.566515+0000", 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.8123999999999999e-05, 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052968999999999998, 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128231 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566515+0000", 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566515+0000", 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566516+0000", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.282e-06 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566521+0000", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0629999999999996e-06 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566534+0000", 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2860999999999999e-05 2024-09-11T05:52:27.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566544+0000", 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9629999999999993e-06 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128235 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.566560+0000", 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2493000000000001e-05, 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054110699999999998, 2024-09-11T05:52:27.596 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128235 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566560+0000", 2024-09-11T05:52:27.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566560+0000", 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566561+0000", 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3600000000000003e-07 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566564+0000", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.14e-06 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566571+0000", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5320000000000003e-06 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566576+0000", 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6979999999999997e-06 2024-09-11T05:52:27.599 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128236 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.566582+0000", 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0523e-05, 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058124999999999995, 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128236 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566582+0000", 2024-09-11T05:52:27.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566582+0000", 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566583+0000", 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3500000000000002e-07 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566585+0000", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566591+0000", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7590000000000003e-06 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566595+0000", 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7910000000000004e-06 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.603 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128238 3.1f 3.9296d2ff (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.566601+0000", 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.739e-06, 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000621811, 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128238 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566601+0000", 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566601+0000", 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566602+0000", 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.22e-07 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566603+0000", 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566609+0000", 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4790000000000004e-06 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.566614+0000", 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6909999999999996e-06 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 11 2024-09-11T05:52:27.607 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:27.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:27.658 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:27.655+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:27.722 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:27.719+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:27.774 INFO:teuthology.orchestra.run.smithi165.stdout:4951 read 0xadf9a1 thru 0xae17ae (0x1e0e bytes) 2024-09-11T05:52:27.782 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:27.782 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:27.782 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:27.782 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:27.782 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:27.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:27.804 INFO:teuthology.orchestra.run.smithi165.stdout:4952 write 0x94c2894 thru 0x94d095c (0xe0c9 bytes) 2024-09-11T05:52:27.816 INFO:teuthology.orchestra.run.smithi165.stdout:4953 write 0x71e7a46 thru 0x71f6a46 (0xf001 bytes) 2024-09-11T05:52:27.822 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:27.822+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:27.825 INFO:teuthology.orchestra.run.smithi165.stdout:4956 read 0x30c282e thru 0x30ca37f (0x7b52 bytes) 2024-09-11T05:52:27.840 INFO:teuthology.orchestra.run.smithi165.stdout:4957 write 0xb754b3b thru 0xb75d207 (0x86cd bytes) 2024-09-11T05:52:27.848 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:27.848 INFO:teuthology.orchestra.run.smithi165.stdout:4960 read 0x3bf0e7f thru 0x3bf9fc3 (0x9145 bytes) 2024-09-11T05:52:27.848 INFO:teuthology.orchestra.run.smithi165.stdout:4961 read 0x3af192d thru 0x3af6dce (0x54a2 bytes) 2024-09-11T05:52:27.848 INFO:teuthology.orchestra.run.smithi165.stdout:4962 write 0x3a729a5 thru 0x3a7463a (0x1c96 bytes) 2024-09-11T05:52:27.853 INFO:teuthology.orchestra.run.smithi165.stdout:4964 writesame 0xe91d28 thru 0xe96161 data_size 0x6(0x443a bytes) 2024-09-11T05:52:27.863 INFO:teuthology.orchestra.run.smithi165.stdout:4965 trunc from 0xb75d208 to 0xd66bfc8 2024-09-11T05:52:27.875 INFO:teuthology.orchestra.run.smithi165.stdout:4966 read 0xb4959e6 thru 0xb495c7a (0x295 bytes) 2024-09-11T05:52:27.876 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:27.876 INFO:teuthology.orchestra.run.smithi165.stdout:4968 write 0x80e0fef thru 0x80e900f (0x8021 bytes) 2024-09-11T05:52:27.882 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:27.882 INFO:teuthology.orchestra.run.smithi165.stdout:4971 punch from 0x1d76c56 to 0x1d79970, (0x2d1a bytes) 2024-09-11T05:52:27.883 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:27.905 INFO:teuthology.orchestra.run.smithi165.stdout:4973 trunc from 0xd66bfc8 to 0x302392a 2024-09-11T05:52:27.924 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:27.919+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:27.930 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.909233667000002, 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:27.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.417910468999999, 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:27.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:27.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:27.940 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.554933695000003, 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:27.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:27.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:27.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.946 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.382356786999999, 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:27.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.382205268, 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:27.951 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:27.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.382162110000003, 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.956 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.382136285999998, 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:27.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:27.965 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.274663017000002, 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:27.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:27.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:27.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90695 3.0 3:03391bbf:::rbd_data.11242d2cfbf4.0000000000000079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.465381938, 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059067149999999999, 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.972 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90695 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464258+0000", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464262+0000", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464277+0000", 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4929999999999999e-05 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464325+0000", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8881000000000002e-05 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464351+0000", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5269000000000002e-05 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464527+0000", 2024-09-11T05:52:27.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017588099999999999 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464774+0000", 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024752500000000001 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.465110+0000", 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033546800000000001 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.467391+0000", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002281047 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470017+0000", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026263079999999999 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470097+0000", 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0584e-05 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470165+0000", 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7655000000000002e-05 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.465239392999997, 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:27.978 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:27.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.118903897999999, 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.986 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:27.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:27.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:27.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.100915750999999, 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:27.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:27.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:27.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.038816466, 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.997 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:27.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:27.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.03870397, 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.001 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.005 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.558759682000002, 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:28.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:28.011 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.557185296, 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:28.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.024376253, 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.017 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:28.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:28.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.534088577999999, 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:28.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:28.029 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128424 3.1f 3:ff4b6949:::rbd_data.112438b89d6.000000000000000d:22 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.360975661, 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060431139999999996, 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.030 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128424 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568667+0000", 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0480000000000001e-06 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568674+0000", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5540000000000003e-06 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568680+0000", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5379999999999999e-06 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574561+0000", 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058812220000000002 2024-09-11T05:52:28.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574638+0000", 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6877000000000003e-05 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574708+0000", 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9503000000000002e-05 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:128869 3.5 e93/84 3:a6f5c53d:::rbd_data.11241be40657.0000000000000004:head v 93'421, mlcod=93'419)", 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.27825209099999998, 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006417191, 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:28.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651390+0000", 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1830000000000001e-06 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653348+0000", 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019573419999999999 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653391+0000", 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3118999999999998e-05 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653414+0000", 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3105999999999999e-05 2024-09-11T05:52:28.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653494+0000", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9722999999999995e-05 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653514+0000", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9962e-05 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657747+0000", 2024-09-11T05:52:28.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042337829999999996 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657805+0000", 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7973000000000002e-05 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.039 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:28.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:4975 write 0xcddd2fc thru 0xcde1c34 (0x4939 bytes) 2024-09-11T05:52:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:4977 read 0x753682f thru 0x7543d1b (0xd4ed bytes) 2024-09-11T05:52:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:4981 write 0x824e9ad thru 0x82563ea (0x7a3e bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4982 read 0x564d7ec thru 0x5653dc6 (0x65db bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4984 read 0x1397d3e thru 0x139fe9f (0x8162 bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4986 write 0x68a02a thru 0x693f7e (0x9f55 bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4987 punch from 0xaee27b3 to 0xaee5ce6, (0x3533 bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4988 punch from 0x5641708 to 0x5641f76, (0x86e bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4989 writesame 0x959fa7f thru 0x95ab99d data_size 0x2f(0xbf1f bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4990 read 0x74cd520 thru 0x74d477f (0x7260 bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4991 punch from 0x904be0c to 0x905a660, (0xe854 bytes) 2024-09-11T05:52:28.043 INFO:teuthology.orchestra.run.smithi165.stdout:4992 writesame 0x318f7b2 thru 0x3197d14 data_size 0x1(0x8563 bytes) 2024-09-11T05:52:28.043 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:28.039+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:28.054 INFO:teuthology.orchestra.run.smithi165.stdout:4993 read 0x2ebef0c thru 0x2ecb4d0 (0xc5c5 bytes) 2024-09-11T05:52:28.057 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.057 INFO:teuthology.orchestra.run.smithi165.stdout:4995 read 0xbe824f3 thru 0xbe82f6c (0xa7a bytes) 2024-09-11T05:52:28.057 INFO:teuthology.orchestra.run.smithi165.stdout:4997 read 0x537f57e thru 0x538cc43 (0xd6c6 bytes) 2024-09-11T05:52:28.057 INFO:teuthology.orchestra.run.smithi165.stdout:4998 write 0x9bf50d3 thru 0x9bfe84b (0x9779 bytes) 2024-09-11T05:52:28.066 INFO:teuthology.orchestra.run.smithi165.stdout:4999 punch from 0x8afe0f9 to 0x8b05cf9, (0x7c00 bytes) 2024-09-11T05:52:28.066 INFO:teuthology.orchestra.run.smithi165.stdout:5001 read 0x682df2c thru 0x682f335 (0x140a bytes) 2024-09-11T05:52:28.066 INFO:teuthology.orchestra.run.smithi165.stdout:5002 write 0xa2b2f52 thru 0xa2b3f01 (0xfb0 bytes) 2024-09-11T05:52:28.074 INFO:teuthology.orchestra.run.smithi165.stdout:5003 read 0xc4b7903 thru 0xc4c0862 (0x8f60 bytes) 2024-09-11T05:52:28.074 INFO:teuthology.orchestra.run.smithi165.stdout:5004 read 0x7fbfa22 thru 0x7fca382 (0xa961 bytes) 2024-09-11T05:52:28.074 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.074 INFO:teuthology.orchestra.run.smithi165.stdout:5006 write 0x89e2a13 thru 0x89e645f (0x3a4d bytes) 2024-09-11T05:52:28.079 INFO:teuthology.orchestra.run.smithi165.stdout:5007 trunc from 0xcde1c35 to 0xc5742f7 2024-09-11T05:52:28.096 INFO:teuthology.orchestra.run.smithi165.stdout:5008 read 0x3492dc5 thru 0x34980c4 (0x5300 bytes) 2024-09-11T05:52:28.096 INFO:teuthology.orchestra.run.smithi165.stdout:5009 write 0x7b2fc70 thru 0x7b3b503 (0xb894 bytes) 2024-09-11T05:52:28.107 INFO:teuthology.orchestra.run.smithi165.stdout:5010 read 0xb9dc967 thru 0xb9ea60d (0xdca7 bytes) 2024-09-11T05:52:28.107 INFO:teuthology.orchestra.run.smithi165.stdout:5011 read 0x9360ad2 thru 0x9367f4b (0x747a bytes) 2024-09-11T05:52:28.109 INFO:teuthology.orchestra.run.smithi165.stdout:5012 read 0x3494266 thru 0x34a2db0 (0xeb4b bytes) 2024-09-11T05:52:28.109 INFO:teuthology.orchestra.run.smithi165.stdout:5013 read 0x77b1b2e thru 0x77bdf38 (0xc40b bytes) 2024-09-11T05:52:28.109 INFO:teuthology.orchestra.run.smithi165.stdout:5014 read 0x776cd1b thru 0x776d2e3 (0x5c9 bytes) 2024-09-11T05:52:28.109 INFO:teuthology.orchestra.run.smithi165.stdout:5015 read 0xc388a38 thru 0xc38adb2 (0x237b bytes) 2024-09-11T05:52:28.109 INFO:teuthology.orchestra.run.smithi165.stdout:5016 read 0xbed1256 thru 0xbedbb74 (0xa91f bytes) 2024-09-11T05:52:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:5017 punch from 0x653722d to 0x6545c84, (0xea57 bytes) 2024-09-11T05:52:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:5018 punch from 0x582b70b to 0x5833411, (0x7d06 bytes) 2024-09-11T05:52:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:5019 read 0x8d2f488 thru 0x8d39742 (0xa2bb bytes) 2024-09-11T05:52:28.110 INFO:teuthology.orchestra.run.smithi165.stdout:5020 punch from 0x289f0cb to 0x28a5ea5, (0x6dda bytes) 2024-09-11T05:52:28.117 INFO:teuthology.orchestra.run.smithi165.stdout:5021 trunc from 0xc5742f7 to 0x964ac10 2024-09-11T05:52:28.142 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:28.142+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:28.148 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.148 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.165 INFO:teuthology.orchestra.run.smithi165.stdout:5026 writesame 0xb66dc0c thru 0xb67710e data_size 0x1(0x9503 bytes) 2024-09-11T05:52:28.179 INFO:teuthology.orchestra.run.smithi165.stdout:5027 trunc from 0xb67710f to 0x54da06b 2024-09-11T05:52:28.197 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:28.197 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:28.197 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:28.197 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:28.206 INFO:teuthology.orchestra.run.smithi165.stdout:5028 punch from 0x3bb90ad to 0x3bb9e2e, (0xd81 bytes) 2024-09-11T05:52:28.206 INFO:teuthology.orchestra.run.smithi165.stdout:5030 read 0xb3acc2 thru 0xb40308 (0x5647 bytes) 2024-09-11T05:52:28.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:28.222 INFO:teuthology.orchestra.run.smithi165.stdout:5031 write 0x660bd18 thru 0x661497b (0x8c64 bytes) 2024-09-11T05:52:28.243 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:28.242+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:28.246 INFO:teuthology.orchestra.run.smithi165.stdout:5032 writesame 0x789af5b thru 0x789c901 data_size 0xb(0x19a7 bytes) 2024-09-11T05:52:28.270 INFO:teuthology.orchestra.run.smithi165.stdout:5033 write 0xea5b0ab thru 0xea61e4d (0x6da3 bytes) 2024-09-11T05:52:28.280 INFO:teuthology.orchestra.run.smithi165.stdout:5035 trunc from 0xea61e4e to 0xea056be 2024-09-11T05:52:28.309 INFO:teuthology.orchestra.run.smithi165.stdout:5036 write 0x9239cfb thru 0x9247669 (0xd96f bytes) 2024-09-11T05:52:28.321 INFO:teuthology.orchestra.run.smithi165.stdout:5038 write 0x9954864 thru 0x995ec61 (0xa3fe bytes) 2024-09-11T05:52:28.332 INFO:teuthology.orchestra.run.smithi165.stdout:5039 punch from 0xd7df307 to 0xd7e47fb, (0x54f4 bytes) 2024-09-11T05:52:28.332 INFO:teuthology.orchestra.run.smithi165.stdout:5040 writesame 0x66e067c thru 0x66e9668 data_size 0x1(0x8fed bytes) 2024-09-11T05:52:28.343 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:28.339+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:28.358 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:28.358 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:28.358 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:28.358 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:28.358 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:28.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:28.373 INFO:teuthology.orchestra.run.smithi165.stdout:5041 punch from 0xde41f5c to 0xde4ddd8, (0xbe7c bytes) 2024-09-11T05:52:28.373 INFO:teuthology.orchestra.run.smithi165.stdout:5042 writesame 0x35a7830 thru 0x35aa49f data_size 0x24(0x2c70 bytes) 2024-09-11T05:52:28.379 INFO:teuthology.orchestra.run.smithi165.stdout:5043 write 0x16569e0 thru 0x165da05 (0x7026 bytes) 2024-09-11T05:52:28.384 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.384 INFO:teuthology.orchestra.run.smithi165.stdout:5045 writesame 0x50e643 thru 0x517648 data_size 0xf(0x9006 bytes) 2024-09-11T05:52:28.388 INFO:teuthology.orchestra.run.smithi165.stdout:5046 write 0x98d1ba6 thru 0x98d7d54 (0x61af bytes) 2024-09-11T05:52:28.401 INFO:teuthology.orchestra.run.smithi165.stdout:5048 trunc from 0xea056be to 0xa529931 2024-09-11T05:52:28.423 INFO:teuthology.orchestra.run.smithi165.stdout:5050 write 0xa6705a2 thru 0xa67ec77 (0xe6d6 bytes) 2024-09-11T05:52:28.432 INFO:teuthology.orchestra.run.smithi165.stdout:5051 read 0x55671f9 thru 0x55765b3 (0xf3bb bytes) 2024-09-11T05:52:28.432 INFO:teuthology.orchestra.run.smithi165.stdout:5052 read 0x5567294 thru 0x556d7ab (0x6518 bytes) 2024-09-11T05:52:28.433 INFO:teuthology.orchestra.run.smithi165.stdout:5053 punch from 0x968af82 to 0x9691117, (0x6195 bytes) 2024-09-11T05:52:28.433 INFO:teuthology.orchestra.run.smithi165.stdout:5054 read 0x8620c56 thru 0x8624bb2 (0x3f5d bytes) 2024-09-11T05:52:28.433 INFO:teuthology.orchestra.run.smithi165.stdout:5055 writesame 0x36b295a thru 0x36ba93c data_size 0x7(0x7fe3 bytes) 2024-09-11T05:52:28.442 INFO:teuthology.orchestra.run.smithi165.stdout:5056 write 0x1cccb6b thru 0x1cd08df (0x3d75 bytes) 2024-09-11T05:52:28.444 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:28.443+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:28.447 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.447 INFO:teuthology.orchestra.run.smithi165.stdout:5058 write 0x94d276e thru 0x94da3b8 (0x7c4b bytes) 2024-09-11T05:52:28.455 INFO:teuthology.orchestra.run.smithi165.stdout:5059 writesame 0x454aff4 thru 0x4552aa6 data_size 0x1(0x7ab3 bytes) 2024-09-11T05:52:28.494 INFO:teuthology.orchestra.run.smithi165.stdout:5061 write 0xe7577e4 thru 0xe75e9bf (0x71dc bytes) 2024-09-11T05:52:28.504 INFO:teuthology.orchestra.run.smithi165.stdout:5062 trunc from 0xe75e9c0 to 0xd4ba5a0 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73347 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1361920~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443598002999998, 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034640307000000002, 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73347 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071187+0000", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071189+0000", 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.843e-06 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071195+0000", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071199+0000", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6410000000000001e-06 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105705+0000", 2024-09-11T05:52:28.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034505909000000001 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105768+0000", 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2871000000000002e-05 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105827+0000", 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8795000000000002e-05 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73352 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1607680~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443561645000003, 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034453226000000003, 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73352 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071225+0000", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071231+0000", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8309999999999997e-06 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071235+0000", 2024-09-11T05:52:28.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.476e-06 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105538+0000", 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034302323000000003 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105603+0000", 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4969999999999996e-05 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105676+0000", 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3435000000000004e-05 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73353 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1581056~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443543398000003, 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035152165999999999, 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73353 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071243+0000", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071249+0000", 2024-09-11T05:52:28.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6869999999999999e-06 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071253+0000", 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106025+0000", 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034771560999999999 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106088+0000", 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2911999999999997e-05 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106393+0000", 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000305352 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73360 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 69632~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:28.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443525145999999, 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034953914000000003, 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73360 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:28.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071260+0000", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7599999999999998e-07 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071262+0000", 2024-09-11T05:52:28.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3769999999999999e-06 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071269+0000", 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5130000000000002e-06 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071273+0000", 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8509999999999998e-06 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105850+0000", 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034576140999999998 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105920+0000", 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0351000000000005e-05 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106213+0000", 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000293305 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.533 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73361 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 278528~18432] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443504574999999, 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035273718000000003, 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.534 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73361 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071282+0000", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071289+0000", 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3090000000000001e-06 2024-09-11T05:52:28.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071293+0000", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7879999999999997e-06 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106234+0000", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034940183999999999 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106301+0000", 2024-09-11T05:52:28.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7638000000000005e-05 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106554+0000", 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025235999999999998 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73362 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 489472~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:28.538 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443484824999999, 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035455391000000003, 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73362 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:28.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0499999999999999e-07 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071302+0000", 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-06 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071308+0000", 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071313+0000", 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6299999999999997e-06 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106625+0000", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035312398000000002 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106693+0000", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7954999999999996e-05 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106755+0000", 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2055000000000001e-05 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73367 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1611776~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443466192999999, 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036019230999999999, 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73367 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.543 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071319+0000", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4400000000000001e-07 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071325+0000", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071331+0000", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071335+0000", 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6319999999999999e-06 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106578+0000", 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035242348999999999 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107279+0000", 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070118999999999997 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107337+0000", 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8646000000000001e-05 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73370 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1888256~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443442244000003, 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035569649000000002, 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.547 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73370 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071343+0000", 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071344+0000", 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071350+0000", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071355+0000", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3850000000000004e-06 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106778+0000", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035423221999999997 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106840+0000", 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106912+0000", 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2181000000000006e-05 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73372 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1329152~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.551 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443423959, 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035712460000000001, 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73372 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071361+0000", 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9499999999999998e-07 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071363+0000", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071369+0000", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071374+0000", 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.78e-06 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106936+0000", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035561622000000001 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106998+0000", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1820000000000001e-05 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107073+0000", 2024-09-11T05:52:28.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5210999999999995e-05 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73374 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1359872~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443404094999998, 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035851052000000001, 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.556 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73374 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:28.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071381+0000", 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5499999999999999e-07 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071383+0000", 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071388+0000", 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7370000000000003e-06 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071393+0000", 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8069999999999999e-06 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107095+0000", 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035701644999999997 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107157+0000", 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107231+0000", 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4828999999999998e-05 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73375 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1333248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.44338501, 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036088839999999997, 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73375 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071400+0000", 2024-09-11T05:52:28.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071402+0000", 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.148e-06 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071409+0000", 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5159999999999997e-06 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071414+0000", 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0000000000000004e-06 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107365+0000", 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035950332000000002 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107429+0000", 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4271999999999997e-05 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107488+0000", 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9215999999999997e-05 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73377 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1363968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443363460999997, 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036963658000000003, 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.565 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73377 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071424+0000", 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.419e-06 2024-09-11T05:52:28.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071431+0000", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071437+0000", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0919999999999997e-06 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108262+0000", 2024-09-11T05:52:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036825113999999999 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108323+0000", 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1292999999999997e-05 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108385+0000", 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1651999999999999e-05 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73380 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1579008~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443341042, 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036798605999999998, 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73380 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071444+0000", 2024-09-11T05:52:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071446+0000", 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2929999999999999e-06 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071453+0000", 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0049999999999998e-06 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071458+0000", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-06 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107511+0000", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036052646000000001 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108165+0000", 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000653934 2024-09-11T05:52:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108242+0000", 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7222e-05 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73382 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1609728~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443319443, 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037077023000000001, 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73382 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.65e-07 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071468+0000", 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071475+0000", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071480+0000", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2020000000000003e-06 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108408+0000", 2024-09-11T05:52:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036928043000000001 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108470+0000", 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1865999999999993e-05 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108542+0000", 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2206999999999995e-05 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73383 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1583104~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443297780999998, 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037204763000000002, 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73383 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071489+0000", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.373e-06 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071495+0000", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8250000000000001e-06 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071500+0000", 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108564+0000", 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037063974 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108628+0000", 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3634999999999996e-05 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108691+0000", 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3647999999999998e-05 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73385 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1613824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443277788000003, 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037327601000000002, 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73385 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8299999999999998e-07 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071509+0000", 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0829999999999998e-06 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071517+0000", 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071523+0000", 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9179999999999999e-06 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108714+0000", 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037191175999999999 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108775+0000", 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108834+0000", 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9447999999999997e-05 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73390 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1890304~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.443254949, 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037487624999999997, 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73390 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071530+0000", 2024-09-11T05:52:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0900000000000002e-07 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071532+0000", 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071538+0000", 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071543+0000", 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9930000000000003e-06 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108856+0000", 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037312788 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108916+0000", 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9570000000000001e-05 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.109017+0000", 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101388 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.673142951000003, 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:52:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.9405264940000002, 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.786200622, 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:52:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:52:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:52:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:52:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:28.609 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:28.609 INFO:teuthology.orchestra.run.smithi165.stdout:5063 writesame 0xd33b59 thru 0xd3e27e data_size 0x16(0xa726 bytes) 2024-09-11T05:52:28.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5065 trunc from 0xd4ba5a0 to 0x9790e8a 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5066 read 0x60b8af1 thru 0x60c0187 (0x7697 bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5067 write 0x35437e2 thru 0x3543b40 (0x35f bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5068 read 0x5311b16 thru 0x531e747 (0xcc32 bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5070 write 0xd69d714 thru 0xd69dfef (0x8dc bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5072 punch from 0x270652e to 0x270cdc7, (0x6899 bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5074 read 0xd365ffd thru 0xd369a39 (0x3a3d bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5075 read 0x6f5a075 thru 0x6f5ba2b (0x19b7 bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5077 punch from 0x9c078f0 to 0x9c15611, (0xdd21 bytes) 2024-09-11T05:52:28.612 INFO:teuthology.orchestra.run.smithi165.stdout:5078 writesame 0x7fcb1bd thru 0x7fd9dc2 data_size 0x1(0xec06 bytes) 2024-09-11T05:52:28.613 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:28.610+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:28.641 INFO:teuthology.orchestra.run.smithi165.stdout:5080 punch from 0x4537c93 to 0x4543ba5, (0xbf12 bytes) 2024-09-11T05:52:28.641 INFO:teuthology.orchestra.run.smithi165.stdout:5083 read 0x5bedb02 thru 0x5bfc68b (0xeb8a bytes) 2024-09-11T05:52:28.642 INFO:teuthology.orchestra.run.smithi165.stdout:5084 read 0x769a2e thru 0x76f2a4 (0x5877 bytes) 2024-09-11T05:52:28.644 INFO:teuthology.orchestra.run.smithi165.stdout:5085 punch from 0xbd3923f to 0xbd39428, (0x1e9 bytes) 2024-09-11T05:52:28.659 INFO:teuthology.orchestra.run.smithi165.stdout:5087 writesame 0xe780ee4 thru 0xe78a331 data_size 0x2(0x944e bytes) 2024-09-11T05:52:28.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:52:28.687 INFO:teuthology.orchestra.run.smithi165.stdout:5088 trunc from 0xe78a332 to 0x29460a3 2024-09-11T05:52:28.711 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:28.710+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:28.735 INFO:teuthology.orchestra.run.smithi165.stdout:5090 writesame 0x2f288c3 thru 0x2f30260 data_size 0x1(0x799e bytes) 2024-09-11T05:52:28.745 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:28.745 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:28.745 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:28.755 INFO:teuthology.orchestra.run.smithi165.stdout:5091 writesame 0x5e71f85 thru 0x5e770af data_size 0x1(0x512b bytes) 2024-09-11T05:52:28.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:28.773 INFO:teuthology.orchestra.run.smithi165.stdout:5092 trunc from 0x5e770b0 to 0xe5bcd79 2024-09-11T05:52:28.786 INFO:teuthology.orchestra.run.smithi165.stdout:5093 writesame 0x67f88bf thru 0x67f9c5f data_size 0x19(0x13a1 bytes) 2024-09-11T05:52:28.807 INFO:teuthology.orchestra.run.smithi165.stdout:5094 trunc from 0xe5bcd79 to 0x854e745 2024-09-11T05:52:28.819 INFO:teuthology.orchestra.run.smithi165.stdout:5097 punch from 0x7f9aff2 to 0x7fa0868, (0x5876 bytes) 2024-09-11T05:52:28.819 INFO:teuthology.orchestra.run.smithi165.stdout:5099 punch from 0x2deaedc to 0x2df6171, (0xb295 bytes) 2024-09-11T05:52:28.819 INFO:teuthology.orchestra.run.smithi165.stdout:5100 punch from 0x343c2bf to 0x343d6c5, (0x1406 bytes) 2024-09-11T05:52:28.819 INFO:teuthology.orchestra.run.smithi165.stdout:5101 read 0x2454d0 thru 0x251e17 (0xc948 bytes) 2024-09-11T05:52:28.819 INFO:teuthology.orchestra.run.smithi165.stdout:5103 punch from 0x20956a9 to 0x20a36a6, (0xdffd bytes) 2024-09-11T05:52:28.820 INFO:teuthology.orchestra.run.smithi165.stdout:5104 read 0x1b42871 thru 0x1b4ebde (0xc36e bytes) 2024-09-11T05:52:28.829 INFO:teuthology.orchestra.run.smithi165.stdout:5106 trunc from 0x854e745 to 0x2168548 2024-09-11T05:52:28.846 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.846 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:28.846+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:28.871 INFO:teuthology.orchestra.run.smithi165.stdout:5108 writesame 0xe5092d7 thru 0xe51544f data_size 0x1(0xc179 bytes) 2024-09-11T05:52:28.872 INFO:teuthology.orchestra.run.smithi165.stdout:5109 write 0x4d7b085 thru 0x4d7db39 (0x2ab5 bytes) 2024-09-11T05:52:28.877 INFO:teuthology.orchestra.run.smithi165.stdout:5110 writesame 0x925964d thru 0x92654fa data_size 0x1(0xbeae bytes) 2024-09-11T05:52:28.906 INFO:teuthology.orchestra.run.smithi165.stdout:5111 read 0x452613f thru 0x45345f3 (0xe4b5 bytes) 2024-09-11T05:52:28.912 INFO:teuthology.orchestra.run.smithi165.stdout:5112 trunc from 0xe515450 to 0xab002a0 2024-09-11T05:52:28.912 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:28.911+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:5115 punch from 0x5d44ccb to 0x5d4fb61, (0xae96 bytes) 2024-09-11T05:52:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:5118 punch from 0xa91bbd5 to 0xa91c854, (0xc7f bytes) 2024-09-11T05:52:28.925 INFO:teuthology.orchestra.run.smithi165.stdout:5120 read 0x976389a thru 0x976add4 (0x753b bytes) 2024-09-11T05:52:28.936 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:28.937 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:28.937 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:28.939 INFO:teuthology.orchestra.run.smithi165.stdout:5122 write 0xb455ec7 thru 0xb46217d (0xc2b7 bytes) 2024-09-11T05:52:28.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:5124 punch from 0x57242cc to 0x572f806, (0xb53a bytes) 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:5126 read 0x96f7fe6 thru 0x9705705 (0xd720 bytes) 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:5129 punch from 0x5319018 to 0x531f000, (0x5fe8 bytes) 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:5130 read 0x5a1a427 thru 0x5a26512 (0xc0ec bytes) 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:28.954 INFO:teuthology.orchestra.run.smithi165.stdout:5132 write 0x7728dde thru 0x77360d2 (0xd2f5 bytes) 2024-09-11T05:52:28.962 INFO:teuthology.orchestra.run.smithi165.stdout:5133 write 0x25863de thru 0x2589e69 (0x3a8c bytes) 2024-09-11T05:52:28.970 INFO:teuthology.orchestra.run.smithi165.stdout:5134 read 0x88f477d thru 0x8903b68 (0xf3ec bytes) 2024-09-11T05:52:28.971 INFO:teuthology.orchestra.run.smithi165.stdout:5135 write 0x82978f6 thru 0x82a551d (0xdc28 bytes) 2024-09-11T05:52:28.981 INFO:teuthology.orchestra.run.smithi165.stdout:5137 punch from 0x1bcfb27 to 0x1bd205c, (0x2535 bytes) 2024-09-11T05:52:28.981 INFO:teuthology.orchestra.run.smithi165.stdout:5139 read 0x4f8582f thru 0x4f87c11 (0x23e3 bytes) 2024-09-11T05:52:28.990 INFO:teuthology.orchestra.run.smithi165.stdout:5140 trunc from 0xb46217e to 0x6e42582 2024-09-11T05:52:29.013 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:29.011+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:29.014 INFO:teuthology.orchestra.run.smithi165.stdout:5143 read 0x1689f90 thru 0x169012d (0x619e bytes) 2024-09-11T05:52:29.038 INFO:teuthology.orchestra.run.smithi165.stdout:5144 write 0xc3ed73a thru 0xc3f186e (0x4135 bytes) 2024-09-11T05:52:29.059 INFO:teuthology.orchestra.run.smithi165.stdout:5145 write 0xe52a11c thru 0xe53112f (0x7014 bytes) 2024-09-11T05:52:29.081 INFO:teuthology.orchestra.run.smithi165.stdout:5146 write 0xeb94c63 thru 0xeb9d326 (0x86c4 bytes) 2024-09-11T05:52:29.090 INFO:teuthology.orchestra.run.smithi165.stdout:5147 write 0x226da9e thru 0x22720cd (0x4630 bytes) 2024-09-11T05:52:29.099 INFO:teuthology.orchestra.run.smithi165.stdout:5148 write 0x224dff1 thru 0x225dee2 (0xfef2 bytes) 2024-09-11T05:52:29.107 INFO:teuthology.orchestra.run.smithi165.stdout:5149 write 0xaf0feb5 thru 0xaf13168 (0x32b4 bytes) 2024-09-11T05:52:29.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:29.115 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.115 INFO:teuthology.orchestra.run.smithi165.stdout:5151 write 0x6d13031 thru 0x6d1734f (0x431f bytes) 2024-09-11T05:52:29.116 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:29.115+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:29.121 INFO:teuthology.orchestra.run.smithi165.stdout:5152 write 0x529eac6 thru 0x52ad6dc (0xec17 bytes) 2024-09-11T05:52:29.130 INFO:teuthology.orchestra.run.smithi165.stdout:5153 punch from 0x273ea4 to 0x27a621, (0x677d bytes) 2024-09-11T05:52:29.130 INFO:teuthology.orchestra.run.smithi165.stdout:5155 writesame 0xcfc0e51 thru 0xcfc2291 data_size 0x11(0x1441 bytes) 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.526758338, 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.136 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:52:29.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.526650434, 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:52:29.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.5265921150000001, 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:29.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.526537993, 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:52:29.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.525547735, 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.525499586, 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:52:29.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:52:29.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.52546732, 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:52:29.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:52:29.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.525437307, 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.167 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.525408375, 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:52:29.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:52:29.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.525366969, 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.175 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.179 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.5217205499999999, 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:29.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:29.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.521637742, 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:52:29.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.52156074, 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:52:29.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.193 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.5214892519999998, 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:52:29.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:52:29.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.52143329, 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.198 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.521394044, 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.5213206960000001, 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:52:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.52129095, 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:52:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:52:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:52:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.5174037330000001, 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.517349345, 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:52:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:52:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:29.228 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5156 punch from 0xdac87c to 0xdb6546, (0x9cca bytes) 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5157 write 0x4593b5c thru 0x45971b1 (0x3656 bytes) 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5159 writesame 0x26497ce thru 0x26505ae data_size 0x1(0x6de1 bytes) 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5160 trunc from 0xeb9d327 to 0x278bd4d 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5161 read 0x142f280 thru 0x143c4e3 (0xd264 bytes) 2024-09-11T05:52:29.230 INFO:teuthology.orchestra.run.smithi165.stdout:5163 write 0x518c4ef thru 0x5196a8a (0xa59c bytes) 2024-09-11T05:52:29.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:29.231 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:29.230+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:29.236 INFO:teuthology.orchestra.run.smithi165.stdout:5165 read 0x2e30e0a thru 0x2e34e8c (0x4083 bytes) 2024-09-11T05:52:29.256 INFO:teuthology.orchestra.run.smithi165.stdout:5166 writesame 0xa82cc1b thru 0xa82dab0 data_size 0x1(0xe96 bytes) 2024-09-11T05:52:29.256 INFO:teuthology.orchestra.run.smithi165.stdout:5167 trunc from 0xa82dab1 to 0xa3623dc 2024-09-11T05:52:29.274 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:29.274 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:29.286 INFO:teuthology.orchestra.run.smithi165.stdout:5168 write 0xcfbe333 thru 0xcfc9538 (0xb206 bytes) 2024-09-11T05:52:29.293 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:29.293 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:29.293 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.293 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.293 INFO:teuthology.orchestra.run.smithi165.stdout:5175 writesame 0xa633ca4 thru 0xa639969 data_size 0x32(0x5cc6 bytes) 2024-09-11T05:52:29.299 INFO:teuthology.orchestra.run.smithi165.stdout:5177 write 0x5e20fc0 thru 0x5e25b48 (0x4b89 bytes) 2024-09-11T05:52:29.302 INFO:teuthology.orchestra.run.smithi165.stdout:5180 write 0x795fb2c thru 0x796e152 (0xe627 bytes) 2024-09-11T05:52:29.308 INFO:teuthology.orchestra.run.smithi165.stdout:5181 punch from 0x919383c to 0x919490c, (0x10d0 bytes) 2024-09-11T05:52:29.308 INFO:teuthology.orchestra.run.smithi165.stdout:5182 writesame 0xcbfcf90 thru 0xcc0bcf6 data_size 0xbb(0xed67 bytes) 2024-09-11T05:52:29.313 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:29.313 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:29.319 INFO:teuthology.orchestra.run.smithi165.stdout:5183 writesame 0xc587706 thru 0xc58a43d data_size 0x4(0x2d38 bytes) 2024-09-11T05:52:29.323 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.323 INFO:teuthology.orchestra.run.smithi165.stdout:5186 read 0xac6d47d thru 0xac79ea1 (0xca25 bytes) 2024-09-11T05:52:29.330 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:29.327+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:29.333 INFO:teuthology.orchestra.run.smithi165.stdout:5187 trunc from 0xcfc9539 to 0x6d1e158 2024-09-11T05:52:29.337 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:29.337 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:29.357 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:29.357 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:29.359 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.379 INFO:teuthology.orchestra.run.smithi165.stdout:5190 write 0xbe772f3 thru 0xbe849ea (0xd6f8 bytes) 2024-09-11T05:52:29.380 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:29.381 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:29.391 INFO:teuthology.orchestra.run.smithi165.stdout:5191 write 0xac25087 thru 0xac252b2 (0x22c bytes) 2024-09-11T05:52:29.396 INFO:teuthology.orchestra.run.smithi165.stdout:5192 read 0x18b7945 thru 0x18bd224 (0x58e0 bytes) 2024-09-11T05:52:29.396 INFO:teuthology.orchestra.run.smithi165.stdout:5193 write 0x884ea29 thru 0x8857d7d (0x9355 bytes) 2024-09-11T05:52:29.403 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:29.404 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:29.404 INFO:teuthology.orchestra.run.smithi165.stdout:5194 punch from 0x1475df9 to 0x147f0f2, (0x92f9 bytes) 2024-09-11T05:52:29.404 INFO:teuthology.orchestra.run.smithi165.stdout:5195 write 0xb84c526 thru 0xb858a02 (0xc4dd bytes) 2024-09-11T05:52:29.410 INFO:teuthology.orchestra.run.smithi165.stdout:5196 punch from 0x5be6216 to 0x5bf21b0, (0xbf9a bytes) 2024-09-11T05:52:29.410 INFO:teuthology.orchestra.run.smithi165.stdout:5197 write 0x1d01355 thru 0x1d0bd50 (0xa9fc bytes) 2024-09-11T05:52:29.413 INFO:teuthology.orchestra.run.smithi165.stdout:5198 read 0x7886747 thru 0x788ebe6 (0x84a0 bytes) 2024-09-11T05:52:29.414 INFO:teuthology.orchestra.run.smithi165.stdout:5199 writesame 0x9e10212 thru 0x9e1fba8 data_size 0xd(0xf997 bytes) 2024-09-11T05:52:29.421 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:29.421 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:29.424 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:29.424 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:29.424 INFO:teuthology.orchestra.run.smithi165.stdout:5200 write 0x1fd82a3 thru 0x1fe0d48 (0x8aa6 bytes) 2024-09-11T05:52:29.431 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:29.430+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:29.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:29.443 INFO:teuthology.orchestra.run.smithi165.stdout:5201 trunc from 0xbe849eb to 0x1642f6c 2024-09-11T05:52:29.478 INFO:teuthology.orchestra.run.smithi165.stdout:5202 read 0xdd2f01 thru 0xdd3c97 (0xd97 bytes) 2024-09-11T05:52:29.478 INFO:teuthology.orchestra.run.smithi165.stdout:5203 read 0x117f011 thru 0x11876bf (0x86af bytes) 2024-09-11T05:52:29.479 INFO:teuthology.orchestra.run.smithi165.stdout:5205 write 0xd786d4 thru 0xd7eb70 (0x649d bytes) 2024-09-11T05:52:29.482 INFO:teuthology.orchestra.run.smithi165.stdout:5206 read 0x398461 thru 0x3a2926 (0xa4c6 bytes) 2024-09-11T05:52:29.482 INFO:teuthology.orchestra.run.smithi165.stdout:5207 read 0x10d0ffb thru 0x10da987 (0x998d bytes) 2024-09-11T05:52:29.483 INFO:teuthology.orchestra.run.smithi165.stdout:5210 punch from 0xfec940 to 0xff9020, (0xc6e0 bytes) 2024-09-11T05:52:29.483 INFO:teuthology.orchestra.run.smithi165.stdout:5213 read 0xb36e52 thru 0xb45b5f (0xed0e bytes) 2024-09-11T05:52:29.484 INFO:teuthology.orchestra.run.smithi165.stdout:5214 punch from 0x9f4ae to 0xaade6, (0xb938 bytes) 2024-09-11T05:52:29.498 INFO:teuthology.orchestra.run.smithi165.stdout:5215 write 0x4fa169a thru 0x4fac8b1 (0xb218 bytes) 2024-09-11T05:52:29.503 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.517 INFO:teuthology.orchestra.run.smithi165.stdout:5218 write 0x87f0edb thru 0x87fe865 (0xd98b bytes) 2024-09-11T05:52:29.526 INFO:teuthology.orchestra.run.smithi165.stdout:5220 punch from 0x26f633f to 0x26fae03, (0x4ac4 bytes) 2024-09-11T05:52:29.526 INFO:teuthology.orchestra.run.smithi165.stdout:5221 read 0x5d5d3 thru 0x6cf1b (0xf949 bytes) 2024-09-11T05:52:29.527 INFO:teuthology.orchestra.run.smithi165.stdout:5222 write 0x4be1b07 thru 0x4beeb4a (0xd044 bytes) 2024-09-11T05:52:29.532 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:29.527+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:29.559 INFO:teuthology.orchestra.run.smithi165.stdout:5223 write 0xde86b28 thru 0xde8ebd6 (0x80af bytes) 2024-09-11T05:52:29.588 INFO:teuthology.orchestra.run.smithi165.stdout:5224 trunc from 0xde8ebd7 to 0x650123b 2024-09-11T05:52:29.612 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:29.612 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:29.616 INFO:teuthology.orchestra.run.smithi165.stdout:5225 punch from 0x2dc2884 to 0x2dd1076, (0xe7f2 bytes) 2024-09-11T05:52:29.616 INFO:teuthology.orchestra.run.smithi165.stdout:5226 read 0x392eaa2 thru 0x3935c7e (0x71dd bytes) 2024-09-11T05:52:29.620 INFO:teuthology.orchestra.run.smithi165.stdout:5227 punch from 0x4535c3b to 0x4540ccb, (0xb090 bytes) 2024-09-11T05:52:29.620 INFO:teuthology.orchestra.run.smithi165.stdout:5228 read 0x413e678 thru 0x414bd4c (0xd6d5 bytes) 2024-09-11T05:52:29.620 INFO:teuthology.orchestra.run.smithi165.stdout:5229 read 0x60e0d2e thru 0x60e9059 (0x832c bytes) 2024-09-11T05:52:29.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:29.632 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:29.631+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:29.641 INFO:teuthology.orchestra.run.smithi165.stdout:5230 write 0xab5f34b thru 0xab6d1ed (0xdea3 bytes) 2024-09-11T05:52:29.653 INFO:teuthology.orchestra.run.smithi165.stdout:5232 read 0x472305f thru 0x47244e3 (0x1485 bytes) 2024-09-11T05:52:29.662 INFO:teuthology.orchestra.run.smithi165.stdout:5233 trunc from 0xab6d1ee to 0x7ac96d1 2024-09-11T05:52:29.665 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:52:29.682 INFO:teuthology.orchestra.run.smithi165.stdout:5234 punch from 0x53e29a1 to 0x53ece63, (0xa4c2 bytes) 2024-09-11T05:52:29.697 INFO:teuthology.orchestra.run.smithi165.stdout:5235 write 0xb2c828b thru 0xb2d77ce (0xf544 bytes) 2024-09-11T05:52:29.703 INFO:teuthology.orchestra.run.smithi165.stdout:5236 read 0x2507e41 thru 0x2513e15 (0xbfd5 bytes) 2024-09-11T05:52:29.703 INFO:teuthology.orchestra.run.smithi165.stdout:5238 write 0xa116d32 thru 0xa1195b8 (0x2887 bytes) 2024-09-11T05:52:29.717 INFO:teuthology.orchestra.run.smithi165.stdout:5239 trunc from 0xb2d77cf to 0x57f2452 2024-09-11T05:52:29.733 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:29.730+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:29.740 INFO:teuthology.orchestra.run.smithi165.stdout:5240 trunc from 0x57f2452 to 0x8068d90 2024-09-11T05:52:29.750 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.750 INFO:teuthology.orchestra.run.smithi165.stdout:5243 write 0x726805f thru 0x7272d5c (0xacfe bytes) 2024-09-11T05:52:29.764 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:29.764 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:29.776 INFO:teuthology.orchestra.run.smithi165.stdout:5244 writesame 0xe7b228e thru 0xe7bfa29 data_size 0x2(0xd79c bytes) 2024-09-11T05:52:29.776 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.776 INFO:teuthology.orchestra.run.smithi165.stdout:5247 write 0x4ff4f60 thru 0x4ffebef (0x9c90 bytes) 2024-09-11T05:52:29.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:29.783 INFO:teuthology.orchestra.run.smithi165.stdout:5248 read 0x4662b29 thru 0x46674e1 (0x49b9 bytes) 2024-09-11T05:52:29.792 INFO:teuthology.orchestra.run.smithi165.stdout:5249 trunc from 0xe7bfa2a to 0x95b3aa8 2024-09-11T05:52:29.820 INFO:teuthology.orchestra.run.smithi165.stdout:5250 write 0x9fe2d20 thru 0x9fe3c0d (0xeee bytes) 2024-09-11T05:52:29.834 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:29.831+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:29.840 INFO:teuthology.orchestra.run.smithi165.stdout:5251 write 0xc4628e8 thru 0xc46d943 (0xb05c bytes) 2024-09-11T05:52:29.870 INFO:teuthology.orchestra.run.smithi165.stdout:5252 trunc from 0xc46d944 to 0x29f43eb 2024-09-11T05:52:29.896 INFO:teuthology.orchestra.run.smithi165.stdout:5253 read 0x6d6777 thru 0x6e27df (0xc069 bytes) 2024-09-11T05:52:29.934 INFO:teuthology.orchestra.run.smithi165.stdout:5255 write 0xc750656 thru 0xc75d65c (0xd007 bytes) 2024-09-11T05:52:29.934 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:29.934+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:29.945 INFO:teuthology.orchestra.run.smithi165.stdout:5256 write 0x329da31 thru 0x32a6dcf (0x939f bytes) 2024-09-11T05:52:29.951 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:29.951 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:29.955 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:29.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:29.970 INFO:teuthology.orchestra.run.smithi165.stdout:5258 trunc from 0xc75d65d to 0x6b1c55b 2024-09-11T05:52:30.007 INFO:teuthology.orchestra.run.smithi165.stdout:5259 trunc from 0x6b1c55b to 0x14e853c 2024-09-11T05:52:30.036 INFO:teuthology.orchestra.run.smithi165.stdout:5260 punch from 0x118302e to 0x11910e0, (0xe0b2 bytes) 2024-09-11T05:52:30.058 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:30.058+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:30.078 INFO:teuthology.orchestra.run.smithi165.stdout:5261 write 0xd5e5328 thru 0xd5f45dc (0xf2b5 bytes) 2024-09-11T05:52:30.091 INFO:teuthology.orchestra.run.smithi165.stdout:5263 read 0x4761a61 thru 0x4767270 (0x5810 bytes) 2024-09-11T05:52:30.091 INFO:teuthology.orchestra.run.smithi165.stdout:5264 punch from 0x48f6a03 to 0x48fb7c5, (0x4dc2 bytes) 2024-09-11T05:52:30.091 INFO:teuthology.orchestra.run.smithi165.stdout:5265 read 0xbd07d71 thru 0xbd11af8 (0x9d88 bytes) 2024-09-11T05:52:30.091 INFO:teuthology.orchestra.run.smithi165.stdout:5266 write 0x1e63b68 thru 0x1e6e3e4 (0xa87d bytes) 2024-09-11T05:52:30.100 INFO:teuthology.orchestra.run.smithi165.stdout:5268 read 0xdcd1ae thru 0xddb3fe (0xe251 bytes) 2024-09-11T05:52:30.101 INFO:teuthology.orchestra.run.smithi165.stdout:5270 write 0xd50ca47 thru 0xd5140bc (0x7676 bytes) 2024-09-11T05:52:30.122 INFO:teuthology.orchestra.run.smithi165.stdout:5271 trunc from 0xd5f45dd to 0x7e333e8 2024-09-11T05:52:30.135 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:30.134+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:30.149 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.150 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.154 INFO:teuthology.orchestra.run.smithi165.stdout:5272 trunc from 0x7e333e8 to 0x5d4e93f 2024-09-11T05:52:30.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:30.168 INFO:teuthology.orchestra.run.smithi165.stdout:5273 read 0x3a2b6f1 thru 0x3a35af7 (0xa407 bytes) 2024-09-11T05:52:30.168 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:30.204 INFO:teuthology.orchestra.run.smithi165.stdout:5275 writesame 0xeb1cb8b thru 0xeb1f494 data_size 0x33(0x290a bytes) 2024-09-11T05:52:30.204 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:30.204 INFO:teuthology.orchestra.run.smithi165.stdout:5277 write 0x4cf8e3e thru 0x4cff3f6 (0x65b9 bytes) 2024-09-11T05:52:30.236 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:30.235+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:30.246 INFO:teuthology.orchestra.run.smithi165.stdout:5279 trunc from 0xeb1f495 to 0x89684f 2024-09-11T05:52:30.275 INFO:teuthology.orchestra.run.smithi165.stdout:5280 read 0x835bd0 thru 0x83c8ff (0x6d30 bytes) 2024-09-11T05:52:30.277 INFO:teuthology.orchestra.run.smithi165.stdout:5281 clone 33 order 25 su 4194304 sc 12 2024-09-11T05:52:30.329 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.329 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.337 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:30.335+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:30.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:30.439 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:30.435+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:30.467 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.467 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:30.539 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:30.535+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:30.597 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.597 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:30.658 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:30.658+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:30.741 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:30.739+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:30.741 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.741 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:30.841 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:30.838+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:30.847 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone32 from 0x89684f (overlap 0x89684f) to 0x1b7b43 2024-09-11T05:52:30.901 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:30.902 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:30.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:30.964 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:30.963+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:30.984 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-09-11T05:52:30.995 INFO:teuthology.orchestra.run.smithi165.stdout:5282 trunc from 0x89684f to 0x521bc0e 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5283 punch from 0x1e48edc to 0x1e535cc, (0xa6f0 bytes) 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5285 punch from 0x3ee2152 to 0x3eea327, (0x81d5 bytes) 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5286 read 0x2528453 thru 0x253645f (0xe00d bytes) 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5287 read 0x42b8ece thru 0x42c17dd (0x8910 bytes) 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5288 read 0x3224d23 thru 0x3227b5a (0x2e38 bytes) 2024-09-11T05:52:31.013 INFO:teuthology.orchestra.run.smithi165.stdout:5289 write 0x4c5ceae thru 0x4c6bb07 (0xec5a bytes) 2024-09-11T05:52:31.020 INFO:teuthology.orchestra.run.smithi165.stdout:5291 read 0x44c6f6d thru 0x44ca201 (0x3295 bytes) 2024-09-11T05:52:31.034 INFO:teuthology.orchestra.run.smithi165.stdout:5292 write 0x5754723 thru 0x575ba13 (0x72f1 bytes) 2024-09-11T05:52:31.042 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.042 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.042 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.043 INFO:teuthology.orchestra.run.smithi165.stdout:5299 punch from 0x2569827 to 0x256b337, (0x1b10 bytes) 2024-09-11T05:52:31.043 INFO:teuthology.orchestra.run.smithi165.stdout:5300 punch from 0x50c8432 to 0x50d3175, (0xad43 bytes) 2024-09-11T05:52:31.043 INFO:teuthology.orchestra.run.smithi165.stdout:5301 punch from 0x11f5b61 to 0x11f9757, (0x3bf6 bytes) 2024-09-11T05:52:31.043 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:31.039+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:31.050 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:31.050 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:31.059 INFO:teuthology.orchestra.run.smithi165.stdout:5302 trunc from 0x575ba14 to 0xd05db61 2024-09-11T05:52:31.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:31.074 INFO:teuthology.orchestra.run.smithi165.stdout:5303 punch from 0xc8ff0c7 to 0xc90ba0a, (0xc943 bytes) 2024-09-11T05:52:31.074 INFO:teuthology.orchestra.run.smithi165.stdout:5304 punch from 0x340ae46 to 0x340e2f4, (0x34ae bytes) 2024-09-11T05:52:31.075 INFO:teuthology.orchestra.run.smithi165.stdout:5305 write 0x35c2d26 thru 0x35c57a8 (0x2a83 bytes) 2024-09-11T05:52:31.110 INFO:teuthology.orchestra.run.smithi165.stdout:5306 punch from 0xeee361 to 0xefd953, (0xf5f2 bytes) 2024-09-11T05:52:31.110 INFO:teuthology.orchestra.run.smithi165.stdout:5307 writesame 0x8b223e7 thru 0x8b28747 data_size 0x13(0x6361 bytes) 2024-09-11T05:52:31.119 INFO:teuthology.orchestra.run.smithi165.stdout:5308 read 0x60ea05 thru 0x61c9a8 (0xdfa4 bytes) 2024-09-11T05:52:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.121 INFO:teuthology.orchestra.run.smithi165.stdout:5313 write 0xbbc133 thru 0xbbd5a3 (0x1471 bytes) 2024-09-11T05:52:31.133 INFO:teuthology.orchestra.run.smithi165.stdout:5314 write 0xb0dd34a thru 0xb0e5853 (0x850a bytes) 2024-09-11T05:52:31.142 INFO:teuthology.orchestra.run.smithi165.stdout:5315 writesame 0xba95b20 thru 0xba9c0bf data_size 0x30(0x65a0 bytes) 2024-09-11T05:52:31.143 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:31.142+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:31.148 INFO:teuthology.orchestra.run.smithi165.stdout:5316 trunc from 0xd05db61 to 0xe74d556 2024-09-11T05:52:31.169 INFO:teuthology.orchestra.run.smithi165.stdout:5317 trunc from 0xe74d556 to 0x8bd52a4 2024-09-11T05:52:31.213 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:31.213 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:31.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:31.244 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:31.242+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:31.345 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:31.342+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:31.390 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:31.390 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:31.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:31.417 INFO:teuthology.orchestra.run.smithi165.stdout:5319 read 0x1300398 thru 0x1307ced (0x7956 bytes) 2024-09-11T05:52:31.421 INFO:teuthology.orchestra.run.smithi165.stdout:5320 trunc from 0x8bd52a4 to 0xb6499ab 2024-09-11T05:52:31.432 INFO:teuthology.orchestra.run.smithi165.stdout:5321 write 0xafe536b thru 0xaff34ba (0xe150 bytes) 2024-09-11T05:52:31.434 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.435 INFO:teuthology.orchestra.run.smithi165.stdout:5324 trunc from 0xb6499ab to 0xba0f88e 2024-09-11T05:52:31.446 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:31.443+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:31.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:31.457 INFO:teuthology.orchestra.run.smithi165.stdout:5325 writesame 0xdbfe624 thru 0xdc07b08 data_size 0x1(0x94e5 bytes) 2024-09-11T05:52:31.458 INFO:teuthology.orchestra.run.smithi165.stdout:5327 write 0xb0443d2 thru 0xb04e73b (0xa36a bytes) 2024-09-11T05:52:31.462 INFO:teuthology.orchestra.run.smithi165.stdout:5328 read 0x137835a thru 0x137a758 (0x23ff bytes) 2024-09-11T05:52:31.462 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.462 INFO:teuthology.orchestra.run.smithi165.stdout:5330 read 0x64ed330 thru 0x64faf8a (0xdc5b bytes) 2024-09-11T05:52:31.463 INFO:teuthology.orchestra.run.smithi165.stdout:5331 punch from 0x7dcf52 to 0x7e8180, (0xb22e bytes) 2024-09-11T05:52:31.471 INFO:teuthology.orchestra.run.smithi165.stdout:5332 trunc from 0xdc07b09 to 0x96131e7 2024-09-11T05:52:31.489 INFO:teuthology.orchestra.run.smithi165.stdout:5335 writesame 0x176b925 thru 0x177592c data_size 0x24(0xa008 bytes) 2024-09-11T05:52:31.499 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.499 INFO:teuthology.orchestra.run.smithi165.stdout:5338 writesame 0x5354dfe thru 0x535fa00 data_size 0x1(0xac03 bytes) 2024-09-11T05:52:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:31.546 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:31.543+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:31.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:31.557 INFO:teuthology.orchestra.run.smithi165.stdout:5339 punch from 0x45c39ae to 0x45d23e2, (0xea34 bytes) 2024-09-11T05:52:31.563 INFO:teuthology.orchestra.run.smithi165.stdout:5340 trunc from 0x96131e7 to 0xd54737a 2024-09-11T05:52:31.573 INFO:teuthology.orchestra.run.smithi165.stdout:5341 write 0x3b6bdd6 thru 0x3b776f3 (0xb91e bytes) 2024-09-11T05:52:31.576 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.577 INFO:teuthology.orchestra.run.smithi165.stdout:5344 read 0x4d01650 thru 0x4d0d8c8 (0xc279 bytes) 2024-09-11T05:52:31.596 INFO:teuthology.orchestra.run.smithi165.stdout:5345 trunc from 0xd54737a to 0x470745 2024-09-11T05:52:31.598 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:31.599 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:31.616 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:31.617 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:31.633 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:31.633 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:31.640 INFO:teuthology.orchestra.run.smithi165.stdout:5347 punch from 0x1bdc1e to 0x1cd927, (0xfd09 bytes) 2024-09-11T05:52:31.647 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:31.643+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:31.664 INFO:teuthology.orchestra.run.smithi165.stdout:5349 write 0xaf31f18 thru 0xaf3a4cc (0x85b5 bytes) 2024-09-11T05:52:31.670 INFO:teuthology.orchestra.run.smithi165.stdout:5350 read 0x7827c4c thru 0x782f56f (0x7924 bytes) 2024-09-11T05:52:31.670 INFO:teuthology.orchestra.run.smithi165.stdout:5351 write 0x8af64d3 thru 0x8afc340 (0x5e6e bytes) 2024-09-11T05:52:31.677 INFO:teuthology.orchestra.run.smithi165.stdout:5355 write 0x46b3efb thru 0x46b6d85 (0x2e8b bytes) 2024-09-11T05:52:31.685 INFO:teuthology.orchestra.run.smithi165.stdout:5357 read 0x1ba327a thru 0x1ba8505 (0x528c bytes) 2024-09-11T05:52:31.685 INFO:teuthology.orchestra.run.smithi165.stdout:5358 write 0x248cb6b thru 0x24952f2 (0x8788 bytes) 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337427736000002, 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:31.686 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:31.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:31.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.690 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337398327000002, 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:31.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:31.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337378270000002, 2024-09-11T05:52:31.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:31.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337336882999999, 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.700 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:31.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337304400999997, 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.705 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:31.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.709 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337284345, 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.710 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:31.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337264843, 2024-09-11T05:52:31.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:31.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337240033999997, 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.719 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:31.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:31.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337220885000001, 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:31.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:31.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337200070999998, 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.728 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337180238000002, 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.733 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:31.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337160298000001, 2024-09-11T05:52:31.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:31.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:31.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:31.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337134177000003, 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.742 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:31.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.337097593000003, 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:31.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.332309793999997, 2024-09-11T05:52:31.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:31.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:31.755 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.332235046999998, 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.756 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:31.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.332162287999999, 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:31.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:31.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.332141368000002, 2024-09-11T05:52:31.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:31.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.990175176999998, 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:31.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:31.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.281735766000001, 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:31.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:31.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:31.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:31.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:31.781 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:31.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:52:31.782 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:31.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:31.782 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-11T05:52:31.782 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T05:52:31.782 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T05:52:31.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5359 writesame 0x4eeb455 thru 0x4ef1796 data_size 0x79(0x6342 bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5360 punch from 0x9218ac6 to 0x921bf15, (0x344f bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5361 punch from 0x3273293 to 0x327aaa4, (0x7811 bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5362 read 0x18b0714 thru 0x18b7df4 (0x76e1 bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5363 punch from 0x89e25b to 0x8a1f26, (0x3ccb bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5365 write 0xa77ee2e thru 0xa785b2b (0x6cfe bytes) 2024-09-11T05:52:31.784 INFO:teuthology.orchestra.run.smithi165.stdout:5366 read 0x19cf089 thru 0x19d0252 (0x11ca bytes) 2024-09-11T05:52:31.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:31.786 INFO:teuthology.orchestra.run.smithi165.stdout:5367 trunc from 0xaf3a4cd to 0x327237f 2024-09-11T05:52:31.786 INFO:teuthology.orchestra.run.smithi165.stdout:5369 trunc from 0x327237f to 0xeaf8c28 2024-09-11T05:52:31.786 INFO:teuthology.orchestra.run.smithi165.stdout:5370 read 0x80db6bf thru 0x80ea1e4 (0xeb26 bytes) 2024-09-11T05:52:31.786 INFO:teuthology.orchestra.run.smithi165.stdout:5371 trunc from 0xeaf8c28 to 0xd3a1f1c 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:5372 write 0xbe90932 thru 0xbe9d7b1 (0xce80 bytes) 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:5374 read 0x32c5246 thru 0x32cfac9 (0xa884 bytes) 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:5375 read 0xaca70c thru 0xad3c40 (0x9535 bytes) 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:5377 read 0x763716b thru 0x763cfff (0x5e95 bytes) 2024-09-11T05:52:31.787 INFO:teuthology.orchestra.run.smithi165.stdout:5381 write 0xd0942bc thru 0xd094be6 (0x92b bytes) 2024-09-11T05:52:31.787 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:31.782+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:31.799 INFO:teuthology.orchestra.run.smithi165.stdout:5382 write 0xdeae4df thru 0xdebb7bc (0xd2de bytes) 2024-09-11T05:52:31.802 INFO:teuthology.orchestra.run.smithi165.stdout:5383 read 0x4c67340 thru 0x4c6db73 (0x6834 bytes) 2024-09-11T05:52:31.803 INFO:teuthology.orchestra.run.smithi165.stdout:5387 write 0x40bbac4 thru 0x40c49a9 (0x8ee6 bytes) 2024-09-11T05:52:31.805 INFO:teuthology.orchestra.run.smithi165.stdout:5388 write 0xc58fb84 thru 0xc59a08e (0xa50b bytes) 2024-09-11T05:52:31.807 INFO:teuthology.orchestra.run.smithi165.stdout:5389 read 0x5aa229e thru 0x5aae31a (0xc07d bytes) 2024-09-11T05:52:31.808 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.808 INFO:teuthology.orchestra.run.smithi165.stdout:5391 read 0x19cc0b7 thru 0x19d432e (0x8278 bytes) 2024-09-11T05:52:31.808 INFO:teuthology.orchestra.run.smithi165.stdout:5393 read 0xc11c703 thru 0xc12373e (0x703c bytes) 2024-09-11T05:52:31.808 INFO:teuthology.orchestra.run.smithi165.stdout:5394 read 0xd6160f3 thru 0xd622427 (0xc335 bytes) 2024-09-11T05:52:31.809 INFO:teuthology.orchestra.run.smithi165.stdout:5395 read 0x7377f73 thru 0x737d873 (0x5901 bytes) 2024-09-11T05:52:31.809 INFO:teuthology.orchestra.run.smithi165.stdout:5396 punch from 0x6098cf1 to 0x609bda9, (0x30b8 bytes) 2024-09-11T05:52:31.811 INFO:teuthology.orchestra.run.smithi165.stdout:5398 trunc from 0xdebb7bd to 0xec8be69 2024-09-11T05:52:31.819 INFO:teuthology.orchestra.run.smithi165.stdout:5400 punch from 0x7f2755c to 0x7f28301, (0xda5 bytes) 2024-09-11T05:52:31.822 INFO:teuthology.orchestra.run.smithi165.stdout:5401 trunc from 0xec8be69 to 0xcd893d8 2024-09-11T05:52:31.833 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.833 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.833 INFO:teuthology.orchestra.run.smithi165.stdout:5404 read 0x4fa2524 thru 0x4fb1205 (0xece2 bytes) 2024-09-11T05:52:31.833 INFO:teuthology.orchestra.run.smithi165.stdout:5405 write 0x78f014b thru 0x78fac76 (0xab2c bytes) 2024-09-11T05:52:31.856 INFO:teuthology.orchestra.run.smithi165.stdout:5406 trunc from 0xcd893d8 to 0x1ff8b4f 2024-09-11T05:52:31.865 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:31.865 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:31.885 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:31.882+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:31.887 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:31.887 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:31.893 INFO:teuthology.orchestra.run.smithi165.stdout:5407 write 0x618d8bb thru 0x6195252 (0x7998 bytes) 2024-09-11T05:52:31.911 INFO:teuthology.orchestra.run.smithi165.stdout:5408 writesame 0xd22f650 thru 0xd237fa6 data_size 0x1(0x8957 bytes) 2024-09-11T05:52:31.913 INFO:teuthology.orchestra.run.smithi165.stdout:5409 write 0x596c483 thru 0x596fd03 (0x3881 bytes) 2024-09-11T05:52:31.917 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.917 INFO:teuthology.orchestra.run.smithi165.stdout:5412 writesame 0xcc4fef9 thru 0xcc5d1f2 data_size 0x16(0xd2fa bytes) 2024-09-11T05:52:31.920 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:31.921 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:31.922 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:31.922 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T05:52:31.922 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:31.932 INFO:teuthology.orchestra.run.smithi165.stdout:5413 trunc from 0xd237fa7 to 0x61b01a3 2024-09-11T05:52:31.947 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.947 INFO:teuthology.orchestra.run.smithi165.stdout:5415 punch from 0x4a857f6 to 0x4a948d6, (0xf0e0 bytes) 2024-09-11T05:52:31.947 INFO:teuthology.orchestra.run.smithi165.stdout:5416 read 0x402047a thru 0x402d666 (0xd1ed bytes) 2024-09-11T05:52:31.949 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:31.950 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:31.951 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:31.952 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:31.952 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:31.952 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:31.968 INFO:teuthology.orchestra.run.smithi165.stdout:5418 write 0xd0a2740 thru 0xd0adb7d (0xb43e bytes) 2024-09-11T05:52:31.971 INFO:teuthology.orchestra.run.smithi165.stdout:5419 read 0xb5eac0a thru 0xb5f200a (0x7401 bytes) 2024-09-11T05:52:31.971 INFO:teuthology.orchestra.run.smithi165.stdout:5420 read 0x1e5b03b thru 0x1e61165 (0x612b bytes) 2024-09-11T05:52:31.971 INFO:teuthology.orchestra.run.smithi165.stdout:5421 write 0x1f0f5ad thru 0x1f159b6 (0x640a bytes) 2024-09-11T05:52:31.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:31.973 INFO:teuthology.orchestra.run.smithi165.stdout:5422 write 0x2415ee0 thru 0x241b20b (0x532c bytes) 2024-09-11T05:52:31.978 INFO:teuthology.orchestra.run.smithi165.stdout:5423 write 0xc42d554 thru 0xc43143c (0x3ee9 bytes) 2024-09-11T05:52:31.982 INFO:teuthology.orchestra.run.smithi165.stdout:5424 read 0x398fd8a thru 0x399a985 (0xabfc bytes) 2024-09-11T05:52:31.982 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.982 INFO:teuthology.orchestra.run.smithi165.stdout:5427 read 0xc6ff679 thru 0xc70af79 (0xb901 bytes) 2024-09-11T05:52:31.983 INFO:teuthology.orchestra.run.smithi165.stdout:5428 punch from 0x1331a07 to 0x133584f, (0x3e48 bytes) 2024-09-11T05:52:31.983 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:31.983 INFO:teuthology.orchestra.run.smithi165.stdout:5431 punch from 0x8fa671e to 0x8fb63ba, (0xfc9c bytes) 2024-09-11T05:52:31.985 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:31.982+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:31.989 INFO:teuthology.orchestra.run.smithi165.stdout:5434 write 0xd757254 thru 0xd76285e (0xb60b bytes) 2024-09-11T05:52:31.995 INFO:teuthology.orchestra.run.smithi165.stdout:5435 trunc from 0xd76285f to 0xc4f61d8 2024-09-11T05:52:32.008 INFO:teuthology.orchestra.run.smithi165.stdout:5436 write 0x796b6da thru 0x7973ee1 (0x8808 bytes) 2024-09-11T05:52:32.012 INFO:teuthology.orchestra.run.smithi165.stdout:5438 write 0x7b91b6b thru 0x7b9d006 (0xb49c bytes) 2024-09-11T05:52:32.015 INFO:teuthology.orchestra.run.smithi165.stdout:5439 read 0x197f339 thru 0x198bd04 (0xc9cc bytes) 2024-09-11T05:52:32.015 INFO:teuthology.orchestra.run.smithi165.stdout:5440 punch from 0x90e6fe5 to 0x90eb579, (0x4594 bytes) 2024-09-11T05:52:32.016 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.016 INFO:teuthology.orchestra.run.smithi165.stdout:5442 read 0x97668ab thru 0x976f84b (0x8fa1 bytes) 2024-09-11T05:52:32.027 INFO:teuthology.orchestra.run.smithi165.stdout:5443 write 0xd402517 thru 0xd40272e (0x218 bytes) 2024-09-11T05:52:32.031 INFO:teuthology.orchestra.run.smithi165.stdout:5444 write 0xd08a396 thru 0xd09a254 (0xfebf bytes) 2024-09-11T05:52:32.036 INFO:teuthology.orchestra.run.smithi165.stdout:5445 trunc from 0xd40272f to 0xe10d611 2024-09-11T05:52:32.045 INFO:teuthology.orchestra.run.smithi165.stdout:5447 read 0x8c2ea73 thru 0x8c3bbc2 (0xd150 bytes) 2024-09-11T05:52:32.045 INFO:teuthology.orchestra.run.smithi165.stdout:5448 punch from 0x608a3b7 to 0x608a57d, (0x1c6 bytes) 2024-09-11T05:52:32.045 INFO:teuthology.orchestra.run.smithi165.stdout:5450 write 0x7df85d1 thru 0x7e05f0a (0xd93a bytes) 2024-09-11T05:52:32.056 INFO:teuthology.orchestra.run.smithi165.stdout:5451 trunc from 0xe10d611 to 0x7bed96d 2024-09-11T05:52:32.072 INFO:teuthology.orchestra.run.smithi165.stdout:5452 read 0x5054d4d thru 0x5062da2 (0xe056 bytes) 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:5453 writesame 0xa9b4598 thru 0xa9b8e0d data_size 0x46(0x4876 bytes) 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:5456 punch from 0x5784f92 to 0x578d1d4, (0x8242 bytes) 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:5457 write 0x28e50d8 thru 0x28eb5d5 (0x64fe bytes) 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:32.369 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:32.370 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:32.370 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:32.370 INFO:teuthology.orchestra.run.smithi165.stdout:5458 write 0xe476273 thru 0xe4763bf (0x14d bytes) 2024-09-11T05:52:32.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:32.371 INFO:teuthology.orchestra.run.smithi165.stdout:5459 punch from 0x1bdf3ec to 0x1be03c8, (0xfdc bytes) 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:5463 read 0x5b1553f thru 0x5b1b7df (0x62a1 bytes) 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:5465 trunc from 0xe4763c0 to 0x673d243 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:5466 write 0xa3a8eaa thru 0xa3b5257 (0xc3ae bytes) 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:5469 read 0x5db0f39 thru 0x5dc0a5c (0xfb24 bytes) 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:32.372 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:32.373 INFO:teuthology.orchestra.run.smithi165.stdout:5470 trunc from 0xa3b5258 to 0xe4d6036 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5471 punch from 0xba5ed65 to 0xba6c53b, (0xd7d6 bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5473 read 0x7532674 thru 0x7539ab5 (0x7442 bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5476 punch from 0x9680d40 to 0x968a790, (0x9a50 bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5477 writesame 0x4da3d55 thru 0x4da7dce data_size 0x3f(0x407a bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5478 read 0xd00fcf6 thru 0xd01d9bf (0xdcca bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5480 punch from 0xd76920c to 0xd773962, (0xa756 bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5481 read 0x3eecc80 thru 0x3ef4912 (0x7c93 bytes) 2024-09-11T05:52:32.374 INFO:teuthology.orchestra.run.smithi165.stdout:5485 trunc from 0xe4d6036 to 0xe1ecc96 2024-09-11T05:52:32.376 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:32.366+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:32.376 INFO:teuthology.orchestra.run.smithi165.stdout:5486 write 0x6f5cec5 thru 0x6f5d7d9 (0x915 bytes) 2024-09-11T05:52:32.376 INFO:teuthology.orchestra.run.smithi165.stdout:5487 read 0xc6fdc5 thru 0xc7a4cd (0xa709 bytes) 2024-09-11T05:52:32.376 INFO:teuthology.orchestra.run.smithi165.stdout:5488 write 0xd19ae13 thru 0xd1a25de (0x77cc bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5491 trunc from 0xe1ecc96 to 0xe1f87e1 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5492 writesame 0xc9b3c17 thru 0xc9b88fa data_size 0x4c(0x4ce4 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5494 punch from 0xca10e27 to 0xca1ae87, (0xa060 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5496 writesame 0x9bd80fa thru 0x9bde9cd data_size 0x2(0x68d4 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5498 write 0x263ce03 thru 0x2640bbb (0x3db9 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5500 punch from 0x3db1a91 to 0x3dbfef6, (0xe465 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5502 trunc from 0xe1f87e1 to 0x8ec3097 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5503 writesame 0x420bf91 thru 0x42101cf data_size 0x1(0x423f bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5504 writesame 0x5f38133 thru 0x5f45c3a data_size 0xa3(0xdb08 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5505 punch from 0x47f8b6e to 0x4803d56, (0xb1e8 bytes) 2024-09-11T05:52:32.377 INFO:teuthology.orchestra.run.smithi165.stdout:5506 trunc from 0x8ec3097 to 0x3ab2740 2024-09-11T05:52:32.378 INFO:teuthology.orchestra.run.smithi165.stdout:5508 trunc from 0x3ab2740 to 0xd5f4b46 2024-09-11T05:52:32.379 INFO:teuthology.orchestra.run.smithi165.stdout:5511 write 0xbe9e33 thru 0xbef1c2 (0x5390 bytes) 2024-09-11T05:52:32.383 INFO:teuthology.orchestra.run.smithi165.stdout:5512 write 0xb8d547d thru 0xb8dfc94 (0xa818 bytes) 2024-09-11T05:52:32.387 INFO:teuthology.orchestra.run.smithi165.stdout:5513 writesame 0x8a920b7 thru 0x8aa0171 data_size 0x3(0xe0bb bytes) 2024-09-11T05:52:32.415 INFO:teuthology.orchestra.run.smithi165.stdout:5514 write 0x1339e13 thru 0x13406f1 (0x68df bytes) 2024-09-11T05:52:32.420 INFO:teuthology.orchestra.run.smithi165.stdout:5515 read 0xb9aa377 thru 0xb9ab291 (0xf1b bytes) 2024-09-11T05:52:32.421 INFO:teuthology.orchestra.run.smithi165.stdout:5517 punch from 0xbe5a6e4 to 0xbe64c53, (0xa56f bytes) 2024-09-11T05:52:32.421 INFO:teuthology.orchestra.run.smithi165.stdout:5518 write 0x859c330 thru 0x85aafe1 (0xecb2 bytes) 2024-09-11T05:52:32.426 INFO:teuthology.orchestra.run.smithi165.stdout:5519 write 0x7c10a4e thru 0x7c15aba (0x506d bytes) 2024-09-11T05:52:32.429 INFO:teuthology.orchestra.run.smithi165.stdout:5520 read 0x95bdf2b thru 0x95c23c5 (0x449b bytes) 2024-09-11T05:52:32.429 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.429 INFO:teuthology.orchestra.run.smithi165.stdout:5522 write 0x1e3819b thru 0x1e3a8d4 (0x273a bytes) 2024-09-11T05:52:32.445 INFO:teuthology.orchestra.run.smithi165.stdout:5523 trunc from 0xd5f4b46 to 0x5aadd03 2024-09-11T05:52:32.463 INFO:teuthology.orchestra.run.smithi165.stdout:5524 write 0x1f32ae8 thru 0x1f3d6f3 (0xac0c bytes) 2024-09-11T05:52:32.466 INFO:teuthology.orchestra.run.smithi165.stdout:5526 write 0xaab527 thru 0xab379d (0x8277 bytes) 2024-09-11T05:52:32.469 INFO:teuthology.orchestra.run.smithi165.stdout:5528 punch from 0x42d93ab to 0x42e153c, (0x8191 bytes) 2024-09-11T05:52:32.470 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:32.470+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:32.477 INFO:teuthology.orchestra.run.smithi165.stdout:5529 trunc from 0x5aadd03 to 0xa36e0de 2024-09-11T05:52:32.487 INFO:teuthology.orchestra.run.smithi165.stdout:5530 write 0x6b60906 thru 0x6b61ca5 (0x13a0 bytes) 2024-09-11T05:52:32.490 INFO:teuthology.orchestra.run.smithi165.stdout:5531 punch from 0x1bee248 to 0x1bee8ff, (0x6b7 bytes) 2024-09-11T05:52:32.490 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.490 INFO:teuthology.orchestra.run.smithi165.stdout:5534 writesame 0x5e4fb42 thru 0x5e5f319 data_size 0x9a(0xf7d8 bytes) 2024-09-11T05:52:32.493 INFO:teuthology.orchestra.run.smithi165.stdout:5536 read 0x84f195e thru 0x84f2bef (0x1292 bytes) 2024-09-11T05:52:32.508 INFO:teuthology.orchestra.run.smithi165.stdout:5538 writesame 0xbd10ea5 thru 0xbd1aae9 data_size 0x69(0x9c45 bytes) 2024-09-11T05:52:32.517 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.493914515, 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.518 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.002591316999997, 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:32.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.139614543, 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.527 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:32.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:32.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.967037634999997, 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.533 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:32.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:32.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.966886115999998, 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.537 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:32.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.966842958000001, 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:32.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:32.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.966817134000003, 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:32.546 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:32.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.550 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.859343865, 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:32.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:32.556 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.049920241000002, 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:32.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.703584746000001, 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.562 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:32.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.685596599, 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:32.572 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.623497314000002, 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.576 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.623384818000002, 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.577 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:32.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.143440529999999, 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.581 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:32.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.141866144000002, 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:32.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.609057101000001, 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:32.593 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.118769426, 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:32.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:32.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:32.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128424 3.1f 3:ff4b6949:::rbd_data.112438b89d6.000000000000000d:22 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.945656509, 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060431139999999996, 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128424 2024-09-11T05:52:32.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:32.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568667+0000", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0480000000000001e-06 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568674+0000", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5540000000000003e-06 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568680+0000", 2024-09-11T05:52:32.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5379999999999999e-06 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574561+0000", 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058812220000000002 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574638+0000", 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6877000000000003e-05 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574708+0000", 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9503000000000002e-05 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:128869 3.5 e93/84 3:a6f5c53d:::rbd_data.11241be40657.0000000000000004:head v 93'421, mlcod=93'419)", 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8629329390000002, 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006417191, 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651390+0000", 2024-09-11T05:52:32.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1830000000000001e-06 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653348+0000", 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019573419999999999 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653391+0000", 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3118999999999998e-05 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653414+0000", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3105999999999999e-05 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653494+0000", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9722999999999995e-05 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653514+0000", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9962e-05 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657747+0000", 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042337829999999996 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657805+0000", 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7973000000000002e-05 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:130898 3.1f 3:fb88de81:::rbd_data.1124fd5e4ce2.0000000000000009:head [truncate 4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.194801892, 2024-09-11T05:52:32.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063299669999999997, 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130898 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319524+0000", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4599999999999996e-06 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319545+0000", 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1322999999999999e-05 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319637+0000", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1605000000000001e-05 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319667+0000", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0000000000000001e-05 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319847+0000", 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017991700000000001 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.320078+0000", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000231506 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.320417+0000", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033864099999999999 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.322204+0000", 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017868059999999999 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325702+0000", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034983079999999999 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325786+0000", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4275000000000002e-05 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325849+0000", 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3126000000000005e-05 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:32.620 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:32.620 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:32.620 INFO:teuthology.orchestra.run.smithi165.stdout:5539 trunc from 0xbd1aaea to 0x12fe216 2024-09-11T05:52:32.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:5542 trunc from 0x12fe216 to 0x7853473 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:5543 punch from 0x3a6e7a8 to 0x3a71ff0, (0x3848 bytes) 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:5544 read 0x6992037 thru 0x6992ad0 (0xa9a bytes) 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:5545 read 0x553671f thru 0x553b940 (0x5222 bytes) 2024-09-11T05:52:32.622 INFO:teuthology.orchestra.run.smithi165.stdout:5546 read 0x109472c thru 0x109fff7 (0xb8cc bytes) 2024-09-11T05:52:32.623 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:32.623 INFO:teuthology.orchestra.run.smithi165.stdout:5549 clone 34 order 20 su 4096 sc 15 2024-09-11T05:52:32.623 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:32.619+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:32.683 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:52:32.721 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:32.718+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:32.797 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:32.797 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:32.797 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:32.798 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:32.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:32.822 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:32.822+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:32.827 INFO:teuthology.orchestra.run.smithi165.stdout:leaving image image_client.0-clone33 intact 2024-09-11T05:52:32.923 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:32.922+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:32.946 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:32.947 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:32.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:33.024 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:33.022+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:33.035 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-09-11T05:52:33.046 INFO:teuthology.orchestra.run.smithi165.stdout:5551 punch from 0x45c14f6 to 0x45d0171, (0xec7b bytes) 2024-09-11T05:52:33.054 INFO:teuthology.orchestra.run.smithi165.stdout:5552 write 0x4e387f5 thru 0x4e4754e (0xed5a bytes) 2024-09-11T05:52:33.102 INFO:teuthology.orchestra.run.smithi165.stdout:5553 write 0x16f10f3 thru 0x16f2353 (0x1261 bytes) 2024-09-11T05:52:33.125 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:33.123+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73347 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1361920~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070398333, 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034640307000000002, 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73347 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071186+0000", 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071187+0000", 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071189+0000", 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.843e-06 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071195+0000", 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-09-11T05:52:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071199+0000", 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6410000000000001e-06 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105705+0000", 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034505909000000001 2024-09-11T05:52:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105768+0000", 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2871000000000002e-05 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105827+0000", 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8795000000000002e-05 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73352 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1607680~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070361974999997, 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034453226000000003, 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73352 2024-09-11T05:52:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071223+0000", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071225+0000", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071231+0000", 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8309999999999997e-06 2024-09-11T05:52:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071235+0000", 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.476e-06 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105538+0000", 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034302323000000003 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105603+0000", 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4969999999999996e-05 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105676+0000", 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3435000000000004e-05 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73353 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1581056~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070343727999997, 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035152165999999999, 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73353 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071241+0000", 2024-09-11T05:52:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071243+0000", 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.916e-06 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071249+0000", 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6869999999999999e-06 2024-09-11T05:52:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071253+0000", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106025+0000", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034771560999999999 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106088+0000", 2024-09-11T05:52:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2911999999999997e-05 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106393+0000", 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000305352 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73360 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 69632~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070325476000001, 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034953914000000003, 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73360 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071259+0000", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071260+0000", 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7599999999999998e-07 2024-09-11T05:52:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071262+0000", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3769999999999999e-06 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071269+0000", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5130000000000002e-06 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071273+0000", 2024-09-11T05:52:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8509999999999998e-06 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105850+0000", 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034576140999999998 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.105920+0000", 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0351000000000005e-05 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106213+0000", 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000293305 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73361 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 278528~18432] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070304905, 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035273718000000003, 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73361 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071280+0000", 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1899999999999998e-07 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071282+0000", 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071289+0000", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3090000000000001e-06 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071293+0000", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7879999999999997e-06 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106234+0000", 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034940183999999999 2024-09-11T05:52:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106301+0000", 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7638000000000005e-05 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106554+0000", 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025235999999999998 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73362 3.18 3:1e619561:::rbd_data.1124767f15a5.0000000000000058:12 [read 489472~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070285155000001, 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035455391000000003, 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73362 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071300+0000", 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0499999999999999e-07 2024-09-11T05:52:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071302+0000", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9939999999999999e-06 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071308+0000", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071313+0000", 2024-09-11T05:52:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6299999999999997e-06 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106625+0000", 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035312398000000002 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106693+0000", 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7954999999999996e-05 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106755+0000", 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2055000000000001e-05 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73367 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1611776~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070266523000001, 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036019230999999999, 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73367 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071318+0000", 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071319+0000", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4400000000000001e-07 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071325+0000", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071331+0000", 2024-09-11T05:52:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071335+0000", 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6319999999999999e-06 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106578+0000", 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035242348999999999 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107279+0000", 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070118999999999997 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107337+0000", 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8646000000000001e-05 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73370 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1888256~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070242573999998, 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035569649000000002, 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73370 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071342+0000", 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071343+0000", 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7399999999999999e-07 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071344+0000", 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.793e-06 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071350+0000", 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071355+0000", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3850000000000004e-06 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106778+0000", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035423221999999997 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106840+0000", 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106912+0000", 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2181000000000006e-05 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73372 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1329152~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070224289000002, 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035712460000000001, 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73372 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071360+0000", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071361+0000", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9499999999999998e-07 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071363+0000", 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019999999999999e-06 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071369+0000", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071374+0000", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.78e-06 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106936+0000", 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035561622000000001 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.106998+0000", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1820000000000001e-05 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107073+0000", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5210999999999995e-05 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73374 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1359872~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:33.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070204425, 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035851052000000001, 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73374 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:33.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071380+0000", 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071381+0000", 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5499999999999999e-07 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071383+0000", 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.874e-06 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071388+0000", 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7370000000000003e-06 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071393+0000", 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8069999999999999e-06 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107095+0000", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035701644999999997 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107157+0000", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1804999999999995e-05 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107231+0000", 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4828999999999998e-05 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73375 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1333248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070185340000002, 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036088839999999997, 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73375 2024-09-11T05:52:33.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071399+0000", 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071400+0000", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5600000000000001e-07 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071402+0000", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.148e-06 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071409+0000", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5159999999999997e-06 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071414+0000", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0000000000000004e-06 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107365+0000", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035950332000000002 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107429+0000", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4271999999999997e-05 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107488+0000", 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9215999999999997e-05 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73377 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1363968~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070163790999999, 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036963658000000003, 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73377 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071421+0000", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6899999999999998e-07 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071424+0000", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.419e-06 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071431+0000", 2024-09-11T05:52:33.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7190000000000001e-06 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071437+0000", 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0919999999999997e-06 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108262+0000", 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036825113999999999 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108323+0000", 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1292999999999997e-05 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108385+0000", 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1651999999999999e-05 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73380 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1579008~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070141372000002, 2024-09-11T05:52:33.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036798605999999998, 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73380 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071443+0000", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071444+0000", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3599999999999999e-07 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071446+0000", 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2929999999999999e-06 2024-09-11T05:52:33.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071453+0000", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0049999999999998e-06 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071458+0000", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0699999999999997e-06 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.107511+0000", 2024-09-11T05:52:33.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036052646000000001 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108165+0000", 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000653934 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108242+0000", 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7222e-05 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73382 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1609728~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070119773000002, 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037077023000000001, 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73382 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071465+0000", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.65e-07 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071468+0000", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3599999999999999e-06 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071475+0000", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071480+0000", 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2020000000000003e-06 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108408+0000", 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036928043000000001 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108470+0000", 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1865999999999993e-05 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108542+0000", 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2206999999999995e-05 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73383 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1583104~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070098111, 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037204763000000002, 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73383 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:33.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071487+0000", 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071489+0000", 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.373e-06 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071495+0000", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8250000000000001e-06 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071500+0000", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9359999999999998e-06 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108564+0000", 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037063974 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108628+0000", 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3634999999999996e-05 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108691+0000", 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3647999999999998e-05 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73385 3.8 3:1349e7fe:::rbd_data.1124a9e466a.0000000000000009:10 [read 1613824~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070078117999998, 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037327601000000002, 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73385 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071507+0000", 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8299999999999998e-07 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071509+0000", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0829999999999998e-06 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071517+0000", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6629999999999993e-06 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071523+0000", 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9179999999999999e-06 2024-09-11T05:52:33.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108714+0000", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037191175999999999 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108775+0000", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108834+0000", 2024-09-11T05:52:33.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9447999999999997e-05 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:73390 3.18 3:1dd90eb8:::rbd_data.1124a9e466a.000000000000000a:10 [read 1890304~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.070055279000002, 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037487624999999997, 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 73390 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071529+0000", 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071530+0000", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0900000000000002e-07 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071532+0000", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071538+0000", 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.071543+0000", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9930000000000003e-06 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108856+0000", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037312788 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.108916+0000", 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9570000000000001e-05 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.109017+0000", 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101388 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.299943280999997, 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.567326824, 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:33.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:52:33.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:52:33.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:52:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:52:33.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:33.232 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.4130009520000009, 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:52:33.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:52:33.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:52:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:33.237 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:33.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5554 read 0x66dfdac thru 0x66e1c03 (0x1e58 bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5556 trunc from 0x7853473 to 0xa037c96 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5559 punch from 0x300d7b to 0x30fb73, (0xedf8 bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5560 punch from 0x9679c62 to 0x967e418, (0x47b6 bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5562 read 0x33c1cdc thru 0x33c4d71 (0x3096 bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5565 trunc from 0xa037c96 to 0xdfa0756 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5568 read 0xa0a9539 thru 0xa0b950b (0xffd3 bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5569 read 0x81aeb6f thru 0x81be42b (0xf8bd bytes) 2024-09-11T05:52:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:5570 writesame 0x87a979 thru 0x889f2f data_size 0x1(0xf5b7 bytes) 2024-09-11T05:52:33.241 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:33.235+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:33.339 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:33.335+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:33.393 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:33.393 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:33.393 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:33.393 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:33.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:33.458 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:33.458+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:33.542 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:33.539+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:33.543 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:33.543 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:33.544 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:33.544 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:33.544 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:33.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:33.611 INFO:teuthology.orchestra.run.smithi165.stdout:5571 clone 35 order 20 su 8192 sc 15 2024-09-11T05:52:33.646 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:33.646+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1170628039999997, 2024-09-11T05:52:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:52:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:52:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1169548999999996, 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:52:33.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:52:33.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1168965809999998, 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.735 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1168424589999999, 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.739 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:52:33.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:52:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.115852201, 2024-09-11T05:52:33.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.747 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1158040519999997, 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1157717859999998, 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.752 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:52:33.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:52:33.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1157417729999999, 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:33.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:52:33.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1157128409999997, 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.761 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:33.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:52:33.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1156714350000003, 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1120250159999996, 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1119422080000003, 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:52:33.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:52:33.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1118652060000001, 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:33.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1117937180000004, 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1117377560000001, 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:52:33.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1116985100000001, 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:52:33.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:52:33.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1116251620000002, 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:52:33.795 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:52:33.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1115954160000001, 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:52:33.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:52:33.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:52:33.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1077081990000002, 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.1076538109999996, 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:33.808 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:52:33.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:52:33.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:33.813 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:33.813 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:33.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:33.814 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:33.811+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:33.915 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:33.911+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:33.922 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone34 from 0xdfa0756 (overlap 0x7853473) to 0x2418fbc 2024-09-11T05:52:34.004 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:34.004 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:34.004 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:34.004 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:34.016 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:34.015+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:34.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:34.117 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:34.114+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:34.148 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:34.148 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:34.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:34.218 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:34.215+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:34.318 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:34.315+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:34.370 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852790156000001, 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:34.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:52:34.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:52:34.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852757982, 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.379 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852734962, 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:52:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852698478000001, 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:52:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:52:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852671207, 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:52:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852648179999999, 2024-09-11T05:52:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:52:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852624465000002, 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:52:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:52:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.85260165, 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852578691000001, 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:52:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852555766000002, 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:52:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:52:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852532052000001, 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852505356000002, 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:52:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852478990000002, 2024-09-11T05:52:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852444295000002, 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.852265007, 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:52:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.849614326000001, 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.849592275999999, 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:52:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:52:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:52:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.849379150000001, 2024-09-11T05:52:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.849348986999999, 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:52:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.849325090000001, 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:52:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:52:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:52:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:34.462 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:34.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:34.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:34.464 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:34.459+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:34.564 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:34.563+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:34.609 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:34.609 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:34.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:34.665 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:34.665 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:34.666 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:34.663+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:34.688 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:34.688 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:34.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:52:34.714 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:34.714 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:34.733 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:34.733 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:34.757 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:34.757 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:34.767 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:34.766+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:34.773 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:34.773 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:34.773 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:34.774 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:34.774 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:34.776 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:34.776 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:34.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:34.798 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:34.798 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:34.817 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:34.817 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:34.878 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:34.878+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:34.969 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:34.967+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:34.970 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:34.971 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:34.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:35.078 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:35.078+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:35.153 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:35.153 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:35.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:35.170 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:35.167+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:35.200 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-09-11T05:52:35.271 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:35.267+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:35.333 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:35.334 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:35.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:35.394 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:35.394+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:35.472 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:35.472 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:35.473 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:35.471+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:35.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:35.519 INFO:teuthology.orchestra.run.smithi165.stdout:5573 trunc from 0xdfa0756 to 0xbf1d06e 2024-09-11T05:52:35.562 INFO:teuthology.orchestra.run.smithi165.stdout:5574 punch from 0x75d2776 to 0x75da484, (0x7d0e bytes) 2024-09-11T05:52:35.562 INFO:teuthology.orchestra.run.smithi165.stdout:5576 writesame 0x5ecf42b thru 0x5ed8167 data_size 0x13(0x8d3d bytes) 2024-09-11T05:52:35.573 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:35.571+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:35.636 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:35.636 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:35.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:35.692 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:35.691+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:35.707 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:52:35.738 INFO:teuthology.orchestra.run.smithi165.stdout:5577 trunc from 0xbf1d06e to 0x5538bd 2024-09-11T05:52:35.775 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:35.774+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:35.860 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:35.860 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:35.875 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:35.874+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:35.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:35.976 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:35.975+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:36.037 INFO:teuthology.orchestra.run.smithi165.stdout:5578 trunc from 0x5538bd to 0xbbce8c5 2024-09-11T05:52:36.044 INFO:teuthology.orchestra.run.smithi165.stdout:5579 punch from 0x447cdb6 to 0x448a65d, (0xd8a7 bytes) 2024-09-11T05:52:36.044 INFO:teuthology.orchestra.run.smithi165.stdout:5580 trunc from 0xbbce8c5 to 0xbbcf76d 2024-09-11T05:52:36.055 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.055 INFO:teuthology.orchestra.run.smithi165.stdout:5583 read 0x4a200c6 thru 0x4a25a7c (0x59b7 bytes) 2024-09-11T05:52:36.055 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.065 INFO:teuthology.orchestra.run.smithi165.stdout:5586 write 0xd7486fa thru 0xd74eb4f (0x6456 bytes) 2024-09-11T05:52:36.072 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.072 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.076 INFO:teuthology.orchestra.run.smithi165.stdout:5587 read 0xb1d42fc thru 0xb1e22d2 (0xdfd7 bytes) 2024-09-11T05:52:36.076 INFO:teuthology.orchestra.run.smithi165.stdout:5588 read 0xc1f0ce8 thru 0xc1ffca2 (0xefbb bytes) 2024-09-11T05:52:36.076 INFO:teuthology.orchestra.run.smithi165.stdout:5590 punch from 0x9daecec to 0x9dbd255, (0xe569 bytes) 2024-09-11T05:52:36.076 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.077 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:36.074+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:36.085 INFO:teuthology.orchestra.run.smithi165.stdout:5592 writesame 0xd98f13f thru 0xd99b729 data_size 0x1(0xc5eb bytes) 2024-09-11T05:52:36.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:36.089 INFO:teuthology.orchestra.run.smithi165.stdout:5593 trunc from 0xd99b72a to 0xc913d6c 2024-09-11T05:52:36.112 INFO:teuthology.orchestra.run.smithi165.stdout:5594 writesame 0x12dd6cf thru 0x12ec965 data_size 0x3(0xf297 bytes) 2024-09-11T05:52:36.134 INFO:teuthology.orchestra.run.smithi165.stdout:5595 write 0xc7a4687 thru 0xc7abecd (0x7847 bytes) 2024-09-11T05:52:36.149 INFO:teuthology.orchestra.run.smithi165.stdout:5597 write 0x29e5eb3 thru 0x29e90d9 (0x3227 bytes) 2024-09-11T05:52:36.158 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.158 INFO:teuthology.orchestra.run.smithi165.stdout:5599 read 0x8374e7c thru 0x8378ebf (0x4044 bytes) 2024-09-11T05:52:36.158 INFO:teuthology.orchestra.run.smithi165.stdout:5600 writesame 0x30d2b0d thru 0x30d52bb data_size 0x1(0x27af bytes) 2024-09-11T05:52:36.169 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.169 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:36.169 INFO:teuthology.orchestra.run.smithi165.stdout:5603 writesame 0x4f406fc thru 0x4f45808 data_size 0x1(0x510d bytes) 2024-09-11T05:52:36.178 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:36.178+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:36.189 INFO:teuthology.orchestra.run.smithi165.stdout:5604 read 0xa1e03ad thru 0xa1eac99 (0xa8ed bytes) 2024-09-11T05:52:36.189 INFO:teuthology.orchestra.run.smithi165.stdout:5607 punch from 0x645d511 to 0x645f1db, (0x1cca bytes) 2024-09-11T05:52:36.190 INFO:teuthology.orchestra.run.smithi165.stdout:5608 read 0x9f010a2 thru 0x9f05b8f (0x4aee bytes) 2024-09-11T05:52:36.190 INFO:teuthology.orchestra.run.smithi165.stdout:5609 writesame 0xa9e5f53 thru 0xa9eeb40 data_size 0x1(0x8bee bytes) 2024-09-11T05:52:36.222 INFO:teuthology.orchestra.run.smithi165.stdout:5610 write 0xa8245d8 thru 0xa830679 (0xc0a2 bytes) 2024-09-11T05:52:36.231 INFO:teuthology.orchestra.run.smithi165.stdout:5611 punch from 0x2370488 to 0x2371b09, (0x1681 bytes) 2024-09-11T05:52:36.231 INFO:teuthology.orchestra.run.smithi165.stdout:5612 read 0x26af54c thru 0x26bb369 (0xbe1e bytes) 2024-09-11T05:52:36.245 INFO:teuthology.orchestra.run.smithi165.stdout:5613 writesame 0xca02c52 thru 0xca0503d data_size 0x2c(0x23ec bytes) 2024-09-11T05:52:36.247 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.247 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.248 INFO:teuthology.orchestra.run.smithi165.stdout:5614 clone 36 order 18 su 2048 sc 9 2024-09-11T05:52:36.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:36.280 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:36.278+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:36.381 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:36.378+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:36.425 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.425 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:36.484 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:36.483+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:36.559 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.559 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:36.616 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:36.615+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:36.684 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:36.682+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:36.742 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.742 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:36.784 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:36.782+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:36.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:36.885 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:36.882+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:36.893 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:36.893 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:36.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:36.921 INFO:teuthology.orchestra.run.smithi165.stdout:leaving image image_client.0-clone35 intact 2024-09-11T05:52:36.937 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:36.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:36.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:52:36.985 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:36.982+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:36.995 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:36.995 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:37.013 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:37.013 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:37.030 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:37.030 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.040 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:58574 3.0 e52/23 3:02ba7c14:::rbd_data.1124767f15a5.0000000000000035:head v 52'191, mlcod=51'189)", 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.017676772000002, 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068250580000000002, 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020407+0000", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020408+0000", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1689999999999999e-06 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020633+0000", 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000225042 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020646+0000", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.269e-05 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020654+0000", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1389999999999995e-06 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020719+0000", 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042999999999994e-05 2024-09-11T05:52:37.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.020736+0000", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7139999999999999e-05 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027176+0000", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064404509999999998 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.027232+0000", 2024-09-11T05:52:37.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5383999999999997e-05 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.526353573999998, 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:52:37.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78428 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,truncate 1835008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:37.049 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.663376800000002, 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059269680000000003, 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78428 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374707+0000", 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374711+0000", 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1330000000000001e-06 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374726+0000", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5557e-05 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374774+0000", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7477e-05 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.374799+0000", 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5236e-05 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375091+0000", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029230400000000001 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375269+0000", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017807500000000001 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.375632+0000", 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036287800000000002 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.377645+0000", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020129150000000001 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380344+0000", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026984790000000002 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380522+0000", 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017848199999999999 2024-09-11T05:52:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:46.380634+0000", 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111432 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84926 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 147456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.490799891999998, 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059768879999999996, 2024-09-11T05:52:37.055 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84926 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547284+0000", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7e-07 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547287+0000", 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6419999999999999e-06 2024-09-11T05:52:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547307+0000", 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0358000000000001e-05 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547327+0000", 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0152e-05 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552857+0000", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055303080000000003 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.552940+0000", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2701000000000007e-05 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553260+0000", 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032035700000000001 2024-09-11T05:52:37.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84927 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 180224~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.490648372999999, 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062248119999999997, 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84927 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547435+0000", 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547436+0000", 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4099999999999998e-07 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547448+0000", 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1824000000000001e-05 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547459+0000", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1532000000000001e-05 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547466+0000", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0149999999999998e-06 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553305+0000", 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058389799999999997 2024-09-11T05:52:37.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553391+0000", 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5925000000000001e-05 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553660+0000", 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000268795 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84928 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 212992~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.490605215000002, 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066856340000000002, 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84928 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.065 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547478+0000", 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547479+0000", 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547482+0000", 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0639999999999998e-06 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547489+0000", 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3880000000000002e-06 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547495+0000", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3920000000000002e-06 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553710+0000", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062152869999999999 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.553798+0000", 2024-09-11T05:52:37.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8259999999999999e-05 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554164+0000", 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036570100000000001 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.490579390999997, 2024-09-11T05:52:37.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:52:37.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:52:37.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.383106122000001, 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.074 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:52:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:52:37.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:37.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.573682497999997, 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:52:37.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:37.086 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.227347002999998, 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:52:37.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.4388.0:94442 3.4 e68/56 ondisk, result = 0)", 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.209358856000001, 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061296450000000004, 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828725+0000", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828726+0000", 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5290000000000001e-06 2024-09-11T05:52:37.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828733+0000", 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828756+0000", 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2299000000000001e-05 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.828769+0000", 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3081000000000001e-05 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834688+0000", 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059187399999999996 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834701+0000", 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465999999999999e-05 2024-09-11T05:52:37.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834854+0000", 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015319699999999999 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.147259570999999, 2024-09-11T05:52:37.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:37.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:52:37.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.147147074999999, 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:52:37.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.106 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106724 3.4 3:253767cd:::rbd_data.11242c12cf9f.0000000000000133:head [set-alloc-hint object_size 262144 write_size 262144,write 67195~5669] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.667202787000001, 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005914323, 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106724 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370881+0000", 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370882+0000", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6300000000000001e-06 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370892+0000", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9289999999999993e-06 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370907+0000", 2024-09-11T05:52:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5191000000000001e-05 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.370915+0000", 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371010+0000", 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5667000000000004e-05 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371134+0000", 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123523 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.371285+0000", 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015105699999999999 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372922+0000", 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001637005 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376693+0000", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037710809999999999 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376751+0000", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7533e-05 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.376795+0000", 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4351000000000003e-05 2024-09-11T05:52:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:106726 3.0 3:075447c7:::rbd_data.11242c12cf9f.0000000000000135:head [set-alloc-hint object_size 262144 write_size 262144,write 66560~6144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.665628400999999, 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064867070000000004, 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.113 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 106726 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372455+0000", 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372456+0000", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1260000000000001e-06 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372466+0000", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9359999999999994e-06 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372482+0000", 2024-09-11T05:52:37.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5679999999999999e-05 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372489+0000", 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5310000000000004e-06 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372558+0000", 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8578999999999995e-05 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372702+0000", 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143664 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.372799+0000", 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7e-05 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.374289+0000", 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014901409999999999 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378843+0000", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045545769999999998 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378898+0000", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4663999999999997e-05 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.378942+0000", 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3809e-05 2024-09-11T05:52:37.118 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:109715 3.0 3:07c036c3:::rbd_data.11242c12cf9f.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.132819357999999, 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061070660000000004, 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109715 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905264+0000", 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905266+0000", 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905275+0000", 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-06 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905297+0000", 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905309+0000", 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2782999999999999e-05 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905384+0000", 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5075999999999997e-05 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905452+0000", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7418999999999999e-05 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.905549+0000", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6817999999999994e-05 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.906861+0000", 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013125190000000001 2024-09-11T05:52:37.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911299+0000", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044379429999999997 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911331+0000", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1492999999999998e-05 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:03.911371+0000", 2024-09-11T05:52:37.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0507999999999999e-05 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:112361 3.0 3:0016fb8a:::rbd_header.11242c12cf9f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.642531683000001, 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219289999999997, 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 112361 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395552+0000", 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395555+0000", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7000000000000002e-06 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395572+0000", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6861999999999999e-05 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395611+0000", 2024-09-11T05:52:37.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9153000000000003e-05 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395630+0000", 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8796000000000001e-05 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395737+0000", 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106697 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-11T05:52:37.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.395746+0000", 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0880000000000001e-06 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398523+0000", 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027771649999999998 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.398685+0000", 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161669 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.399312+0000", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062753899999999998 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.400672+0000", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013597100000000001 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401361+0000", 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000688969 2024-09-11T05:52:37.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401454+0000", 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2391000000000004e-05 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:04.401574+0000", 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012019 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:128424 3.1f 3:ff4b6949:::rbd_data.112438b89d6.000000000000000d:22 [read 1015808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.4694187660000004, 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060431139999999996, 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 128424 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568665+0000", 2024-09-11T05:52:37.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.72e-07 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568667+0000", 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0480000000000001e-06 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568674+0000", 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5540000000000003e-06 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.568680+0000", 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5379999999999999e-06 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574561+0000", 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058812220000000002 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574638+0000", 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6877000000000003e-05 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.574708+0000", 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9503000000000002e-05 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:128869 3.5 e93/84 3:a6f5c53d:::rbd_data.11241be40657.0000000000000004:head v 93'421, mlcod=93'419)", 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.3866951959999998, 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006417191, 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.136 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651388+0000", 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.651390+0000", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1830000000000001e-06 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653348+0000", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019573419999999999 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653391+0000", 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3118999999999998e-05 2024-09-11T05:52:37.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653414+0000", 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3105999999999999e-05 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653494+0000", 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9722999999999995e-05 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.653514+0000", 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9962e-05 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657747+0000", 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042337829999999996 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:27.657805+0000", 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7973000000000002e-05 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.140 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:130898 3.1f 3:fb88de81:::rbd_data.1124fd5e4ce2.0000000000000009:head [truncate 4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7185641489999997, 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063299669999999997, 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.141 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 130898 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319519+0000", 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319524+0000", 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4599999999999996e-06 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319545+0000", 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1322999999999999e-05 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319637+0000", 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1605000000000001e-05 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319667+0000", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0000000000000001e-05 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.319847+0000", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017991700000000001 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.320078+0000", 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000231506 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.320417+0000", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033864099999999999 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.322204+0000", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017868059999999999 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325702+0000", 2024-09-11T05:52:37.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034983079999999999 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325786+0000", 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4275000000000002e-05 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:32.325849+0000", 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3126000000000005e-05 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.147 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:37.147 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:37.148 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:37.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:37.149 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:37.149 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:37.150 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:37.150 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:37.150 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:37.150 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:37.151 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:37.151 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:37.151 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:37.146+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:37.211 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-09-11T05:52:37.256 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:37.251+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:37.263 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:52:37.263 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:52:37.281 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-11T05:52:36.712322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1037169982,"num_objects":1709,"num_object_clones":533,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":643,"num_read":181810,"num_read_kb":504939,"num_write":18555,"num_write_kb":296580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2181,"num_bytes_recovered":87811982,"num_keys_recovered":328,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14055,"ondisk_log_size":14055,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":529868,"kb_used_data":277516,"kb_used_omap":9,"kb_used_meta":252214,"kb_avail":565701172,"statfs":{"total":579820584960,"available":579278000128,"internally_reserved":0,"allocated":284176384,"data_stored":352406151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":258267814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13263133,"num_objects":106,"num_object_clones":20,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":56,"num_read":16178,"num_read_kb":27406,"num_write":927,"num_write_kb":24237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.195229"},"pg_stats":[{"pgid":"3.1f","version":"97'420","reported_seq":2478,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.316904+0000","last_change":"2024-09-11T05:52:23.537715+0000","last_active":"2024-09-11T05:52:34.316904+0000","last_peered":"2024-09-11T05:52:34.316904+0000","last_clean":"2024-09-11T05:52:34.316904+0000","last_became_active":"2024-09-11T05:52:23.536523+0000","last_became_peered":"2024-09-11T05:52:23.536523+0000","last_unstale":"2024-09-11T05:52:34.316904+0000","last_undegraded":"2024-09-11T05:52:34.316904+0000","last_fullsized":"2024-09-11T05:52:34.316904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:39:36.072164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090687999999999999,"stat_sum":{"num_bytes":47047138,"num_objects":71,"num_object_clones":26,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":24,"num_read":2147,"num_read_kb":6957,"num_write":504,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":5157352,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"96'296","reported_seq":2057,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.764535+0000","last_change":"2024-09-11T05:51:07.506336+0000","last_active":"2024-09-11T05:52:33.764535+0000","last_peered":"2024-09-11T05:52:33.764535+0000","last_clean":"2024-09-11T05:52:33.764535+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:52:33.764535+0000","last_undegraded":"2024-09-11T05:52:33.764535+0000","last_fullsized":"2024-09-11T05:52:33.764535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:37.625546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059150900000000005,"stat_sum":{"num_bytes":28763597,"num_objects":47,"num_object_clones":11,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":20,"num_read":3038,"num_read_kb":7785,"num_write":337,"num_write_kb":4219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"97'204","reported_seq":3233,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.562632+0000","last_change":"2024-09-11T05:52:17.641069+0000","last_active":"2024-09-11T05:52:34.562632+0000","last_peered":"2024-09-11T05:52:34.562632+0000","last_clean":"2024-09-11T05:52:34.562632+0000","last_became_active":"2024-09-11T05:52:17.638753+0000","last_became_peered":"2024-09-11T05:52:17.638753+0000","last_unstale":"2024-09-11T05:52:34.562632+0000","last_undegraded":"2024-09-11T05:52:34.562632+0000","last_fullsized":"2024-09-11T05:52:34.562632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:42.654208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097427899999999999,"stat_sum":{"num_bytes":28030295,"num_objects":49,"num_object_clones":15,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":19,"num_read":2743,"num_read_kb":13886,"num_write":332,"num_write_kb":7381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"97'368","reported_seq":3169,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.223412+0000","last_change":"2024-09-11T05:52:17.640186+0000","last_active":"2024-09-11T05:52:34.223412+0000","last_peered":"2024-09-11T05:52:34.223412+0000","last_clean":"2024-09-11T05:52:34.223412+0000","last_became_active":"2024-09-11T05:52:17.637949+0000","last_became_peered":"2024-09-11T05:52:17.637949+0000","last_unstale":"2024-09-11T05:52:34.223412+0000","last_undegraded":"2024-09-11T05:52:34.223412+0000","last_fullsized":"2024-09-11T05:52:34.223412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:11.754057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491359,"stat_sum":{"num_bytes":8427572,"num_objects":41,"num_object_clones":15,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":13,"num_read":3694,"num_read_kb":8358,"num_write":496,"num_write_kb":5127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"97'435","reported_seq":7095,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.834594+0000","last_change":"2024-09-11T05:52:23.539454+0000","last_active":"2024-09-11T05:52:34.834594+0000","last_peered":"2024-09-11T05:52:34.834594+0000","last_clean":"2024-09-11T05:52:34.834594+0000","last_became_active":"2024-09-11T05:52:23.535205+0000","last_became_peered":"2024-09-11T05:52:23.535205+0000","last_unstale":"2024-09-11T05:52:34.834594+0000","last_undegraded":"2024-09-11T05:52:34.834594+0000","last_fullsized":"2024-09-11T05:52:34.834594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:30.973991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002297412,"stat_sum":{"num_bytes":43881751,"num_objects":48,"num_object_clones":13,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":16,"num_read":7568,"num_read_kb":34254,"num_write":574,"num_write_kb":10677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":2306591,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"97'296","reported_seq":7648,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.781026+0000","last_change":"2024-09-11T05:51:07.504374+0000","last_active":"2024-09-11T05:52:34.781026+0000","last_peered":"2024-09-11T05:52:34.781026+0000","last_clean":"2024-09-11T05:52:34.781026+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:52:34.781026+0000","last_undegraded":"2024-09-11T05:52:34.781026+0000","last_fullsized":"2024-09-11T05:52:34.781026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:38.664575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052666400000000004,"stat_sum":{"num_bytes":30480872,"num_objects":61,"num_object_clones":19,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":29,"num_read":7365,"num_read_kb":13269,"num_write":391,"num_write_kb":12065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"97'545","reported_seq":3667,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.227242+0000","last_change":"2024-09-11T05:52:08.536765+0000","last_active":"2024-09-11T05:52:34.227242+0000","last_peered":"2024-09-11T05:52:34.227242+0000","last_clean":"2024-09-11T05:52:34.227242+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:52:34.227242+0000","last_undegraded":"2024-09-11T05:52:34.227242+0000","last_fullsized":"2024-09-11T05:52:34.227242+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:24:48.339875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094797800000000004,"stat_sum":{"num_bytes":15699632,"num_objects":64,"num_object_clones":18,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":26,"num_read":4623,"num_read_kb":9514,"num_write":767,"num_write_kb":5846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"96'338","reported_seq":6863,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.764199+0000","last_change":"2024-09-11T05:52:23.532280+0000","last_active":"2024-09-11T05:52:33.764199+0000","last_peered":"2024-09-11T05:52:33.764199+0000","last_clean":"2024-09-11T05:52:33.764199+0000","last_became_active":"2024-09-11T05:52:23.529775+0000","last_became_peered":"2024-09-11T05:52:23.529775+0000","last_unstale":"2024-09-11T05:52:33.764199+0000","last_undegraded":"2024-09-11T05:52:33.764199+0000","last_fullsized":"2024-09-11T05:52:33.764199+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:54:28.877728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001728385,"stat_sum":{"num_bytes":34630106,"num_objects":58,"num_object_clones":19,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":15,"num_read":7535,"num_read_kb":25992,"num_write":401,"num_write_kb":11059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":3756409,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"96'309","reported_seq":2894,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.808352+0000","last_change":"2024-09-11T05:52:24.540728+0000","last_active":"2024-09-11T05:52:33.808352+0000","last_peered":"2024-09-11T05:52:33.808352+0000","last_clean":"2024-09-11T05:52:33.808352+0000","last_became_active":"2024-09-11T05:52:24.538227+0000","last_became_peered":"2024-09-11T05:52:24.538227+0000","last_unstale":"2024-09-11T05:52:33.808352+0000","last_undegraded":"2024-09-11T05:52:33.808352+0000","last_fullsized":"2024-09-11T05:52:33.808352+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:54:56.797236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024468200000000002,"stat_sum":{"num_bytes":18228742,"num_objects":51,"num_object_clones":16,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":19,"num_read":3287,"num_read_kb":6424,"num_write":402,"num_write_kb":6425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":2921918,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"97'224","reported_seq":8723,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.644806+0000","last_change":"2024-09-11T05:52:17.637099+0000","last_active":"2024-09-11T05:52:34.644806+0000","last_peered":"2024-09-11T05:52:34.644806+0000","last_clean":"2024-09-11T05:52:34.644806+0000","last_became_active":"2024-09-11T05:52:17.635558+0000","last_became_peered":"2024-09-11T05:52:17.635558+0000","last_unstale":"2024-09-11T05:52:34.644806+0000","last_undegraded":"2024-09-11T05:52:34.644806+0000","last_fullsized":"2024-09-11T05:52:34.644806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:32:31.352812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074373500000000003,"stat_sum":{"num_bytes":38571271,"num_objects":52,"num_object_clones":15,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":24,"num_read":8214,"num_read_kb":23691,"num_write":295,"num_write_kb":15048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"97'358","reported_seq":7114,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.871831+0000","last_change":"2024-09-11T05:51:07.537306+0000","last_active":"2024-09-11T05:52:34.871831+0000","last_peered":"2024-09-11T05:52:34.871831+0000","last_clean":"2024-09-11T05:52:34.871831+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:52:34.871831+0000","last_undegraded":"2024-09-11T05:52:34.871831+0000","last_fullsized":"2024-09-11T05:52:34.871831+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:45.254126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001795668,"stat_sum":{"num_bytes":48001581,"num_objects":42,"num_object_clones":11,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":17,"num_read":7685,"num_read_kb":10018,"num_write":361,"num_write_kb":8023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"97'721","reported_seq":5156,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.554111+0000","last_change":"2024-09-11T05:51:07.504346+0000","last_active":"2024-09-11T05:52:34.554111+0000","last_peered":"2024-09-11T05:52:34.554111+0000","last_clean":"2024-09-11T05:52:34.554111+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:52:34.554111+0000","last_undegraded":"2024-09-11T05:52:34.554111+0000","last_fullsized":"2024-09-11T05:52:34.554111+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:16:55.356607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":68947276,"num_objects":72,"num_object_clones":26,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":22,"num_read":6809,"num_read_kb":20789,"num_write":963,"num_write_kb":13939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.711095+0000","last_change":"2024-09-11T05:51:12.371188+0000","last_active":"2024-09-11T05:52:33.711095+0000","last_peered":"2024-09-11T05:52:33.711095+0000","last_clean":"2024-09-11T05:52:33.711095+0000","last_became_active":"2024-09-11T05:51:12.370422+0000","last_became_peered":"2024-09-11T05:51:12.370422+0000","last_unstale":"2024-09-11T05:52:33.711095+0000","last_undegraded":"2024-09-11T05:52:33.711095+0000","last_fullsized":"2024-09-11T05:52:33.711095+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:20.418215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"97'566","reported_seq":4344,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.620361+0000","last_change":"2024-09-11T05:52:23.530194+0000","last_active":"2024-09-11T05:52:34.620361+0000","last_peered":"2024-09-11T05:52:34.620361+0000","last_clean":"2024-09-11T05:52:34.620361+0000","last_became_active":"2024-09-11T05:52:23.528940+0000","last_became_peered":"2024-09-11T05:52:23.528940+0000","last_unstale":"2024-09-11T05:52:34.620361+0000","last_undegraded":"2024-09-11T05:52:34.620361+0000","last_fullsized":"2024-09-11T05:52:34.620361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:05:22.758793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001678494,"stat_sum":{"num_bytes":23946167,"num_objects":57,"num_object_clones":18,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":23,"num_read":4798,"num_read_kb":11126,"num_write":794,"num_write_kb":6867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":7693928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.835776+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:52:33.835776+0000","last_peered":"2024-09-11T05:52:33.835776+0000","last_clean":"2024-09-11T05:52:33.835776+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:52:33.835776+0000","last_undegraded":"2024-09-11T05:52:33.835776+0000","last_fullsized":"2024-09-11T05:52:33.835776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"93'359","reported_seq":2178,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.764744+0000","last_change":"2024-09-11T05:51:07.506800+0000","last_active":"2024-09-11T05:52:33.764744+0000","last_peered":"2024-09-11T05:52:33.764744+0000","last_clean":"2024-09-11T05:52:33.764744+0000","last_became_active":"2024-09-11T05:51:05.516806+0000","last_became_peered":"2024-09-11T05:51:05.516806+0000","last_unstale":"2024-09-11T05:52:33.764744+0000","last_undegraded":"2024-09-11T05:52:33.764744+0000","last_fullsized":"2024-09-11T05:52:33.764744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:22.256494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093160300000000003,"stat_sum":{"num_bytes":10309028,"num_objects":38,"num_object_clones":11,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":17,"num_read":2681,"num_read_kb":8102,"num_write":546,"num_write_kb":4333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":173,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.710783+0000","last_change":"2024-09-11T05:52:20.524063+0000","last_active":"2024-09-11T05:52:33.710783+0000","last_peered":"2024-09-11T05:52:33.710783+0000","last_clean":"2024-09-11T05:52:33.710783+0000","last_became_active":"2024-09-11T05:52:20.523762+0000","last_became_peered":"2024-09-11T05:52:20.523762+0000","last_unstale":"2024-09-11T05:52:33.710783+0000","last_undegraded":"2024-09-11T05:52:33.710783+0000","last_fullsized":"2024-09-11T05:52:33.710783+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:45:23.710067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"97'289","reported_seq":5168,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.218229+0000","last_change":"2024-09-11T05:52:08.530178+0000","last_active":"2024-09-11T05:52:34.218229+0000","last_peered":"2024-09-11T05:52:34.218229+0000","last_clean":"2024-09-11T05:52:34.218229+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:52:34.218229+0000","last_undegraded":"2024-09-11T05:52:34.218229+0000","last_fullsized":"2024-09-11T05:52:34.218229+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:03.485928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001114744,"stat_sum":{"num_bytes":21365753,"num_objects":31,"num_object_clones":12,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":13,"num_read":5140,"num_read_kb":22900,"num_write":298,"num_write_kb":6141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.836006+0000","last_change":"2024-09-11T05:52:20.520188+0000","last_active":"2024-09-11T05:52:33.836006+0000","last_peered":"2024-09-11T05:52:33.836006+0000","last_clean":"2024-09-11T05:52:33.836006+0000","last_became_active":"2024-09-11T05:52:20.519178+0000","last_became_peered":"2024-09-11T05:52:20.519178+0000","last_unstale":"2024-09-11T05:52:33.836006+0000","last_undegraded":"2024-09-11T05:52:33.836006+0000","last_fullsized":"2024-09-11T05:52:33.836006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:03.835266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"95'469","reported_seq":5190,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.823926+0000","last_change":"2024-09-11T05:52:18.643055+0000","last_active":"2024-09-11T05:52:33.823926+0000","last_peered":"2024-09-11T05:52:33.823926+0000","last_clean":"2024-09-11T05:52:33.823926+0000","last_became_active":"2024-09-11T05:52:18.641130+0000","last_became_peered":"2024-09-11T05:52:18.641130+0000","last_unstale":"2024-09-11T05:52:33.823926+0000","last_undegraded":"2024-09-11T05:52:33.823926+0000","last_fullsized":"2024-09-11T05:52:33.823926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:41:50.909684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012814619999999999,"stat_sum":{"num_bytes":53410857,"num_objects":52,"num_object_clones":17,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":18,"num_read":7011,"num_read_kb":20770,"num_write":509,"num_write_kb":10418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"97'598","reported_seq":3020,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.440282+0000","last_change":"2024-09-11T05:52:24.540558+0000","last_active":"2024-09-11T05:52:34.440282+0000","last_peered":"2024-09-11T05:52:34.440282+0000","last_clean":"2024-09-11T05:52:34.440282+0000","last_became_active":"2024-09-11T05:52:24.538605+0000","last_became_peered":"2024-09-11T05:52:24.538605+0000","last_unstale":"2024-09-11T05:52:34.440282+0000","last_undegraded":"2024-09-11T05:52:34.440282+0000","last_fullsized":"2024-09-11T05:52:34.440282+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:22.283963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096617199999999995,"stat_sum":{"num_bytes":10574521,"num_objects":48,"num_object_clones":14,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":18,"num_read":4295,"num_read_kb":9391,"num_write":880,"num_write_kb":5966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":1853425,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":64,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.835458+0000","last_change":"2024-09-11T05:52:14.615928+0000","last_active":"2024-09-11T05:52:33.835458+0000","last_peered":"2024-09-11T05:52:33.835458+0000","last_clean":"2024-09-11T05:52:33.835458+0000","last_became_active":"2024-09-11T05:52:14.609394+0000","last_became_peered":"2024-09-11T05:52:14.609394+0000","last_unstale":"2024-09-11T05:52:33.835458+0000","last_undegraded":"2024-09-11T05:52:33.835458+0000","last_fullsized":"2024-09-11T05:52:33.835458+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:25.789248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"97'483","reported_seq":2631,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.850235+0000","last_change":"2024-09-11T05:52:08.529630+0000","last_active":"2024-09-11T05:52:34.850235+0000","last_peered":"2024-09-11T05:52:34.850235+0000","last_clean":"2024-09-11T05:52:34.850235+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:52:34.850235+0000","last_undegraded":"2024-09-11T05:52:34.850235+0000","last_fullsized":"2024-09-11T05:52:34.850235+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:24:03.313089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008339,"stat_sum":{"num_bytes":19948150,"num_objects":52,"num_object_clones":15,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":20,"num_read":3817,"num_read_kb":9387,"num_write":558,"num_write_kb":5079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.698160+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:52:33.698160+0000","last_peered":"2024-09-11T05:52:33.698160+0000","last_clean":"2024-09-11T05:52:33.698160+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:52:33.698160+0000","last_undegraded":"2024-09-11T05:52:33.698160+0000","last_fullsized":"2024-09-11T05:52:33.698160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'519","reported_seq":4917,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.839125+0000","last_change":"2024-09-11T05:52:24.533966+0000","last_active":"2024-09-11T05:52:34.839125+0000","last_peered":"2024-09-11T05:52:34.839125+0000","last_clean":"2024-09-11T05:52:34.839125+0000","last_became_active":"2024-09-11T05:52:24.533192+0000","last_became_peered":"2024-09-11T05:52:24.533192+0000","last_unstale":"2024-09-11T05:52:34.839125+0000","last_undegraded":"2024-09-11T05:52:34.839125+0000","last_fullsized":"2024-09-11T05:52:34.839125+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:21.842881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093636299999999995,"stat_sum":{"num_bytes":34040948,"num_objects":71,"num_object_clones":21,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":32,"num_read":7299,"num_read_kb":19042,"num_write":635,"num_write_kb":8778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":6497626,"num_keys_recovered":59,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":178,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.709142+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:52:33.709142+0000","last_peered":"2024-09-11T05:52:33.709142+0000","last_clean":"2024-09-11T05:52:33.709142+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:52:33.709142+0000","last_undegraded":"2024-09-11T05:52:33.709142+0000","last_fullsized":"2024-09-11T05:52:33.709142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"97'355","reported_seq":6445,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.929879+0000","last_change":"2024-09-11T05:52:17.638710+0000","last_active":"2024-09-11T05:52:33.929879+0000","last_peered":"2024-09-11T05:52:33.929879+0000","last_clean":"2024-09-11T05:52:33.929879+0000","last_became_active":"2024-09-11T05:52:17.637965+0000","last_became_peered":"2024-09-11T05:52:17.637965+0000","last_unstale":"2024-09-11T05:52:33.929879+0000","last_undegraded":"2024-09-11T05:52:33.929879+0000","last_fullsized":"2024-09-11T05:52:33.929879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:24.785302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000942811,"stat_sum":{"num_bytes":42647988,"num_objects":55,"num_object_clones":19,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":19,"num_read":6321,"num_read_kb":17515,"num_write":491,"num_write_kb":13286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":51,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.698189+0000","last_change":"2024-09-11T05:52:14.618897+0000","last_active":"2024-09-11T05:52:33.698189+0000","last_peered":"2024-09-11T05:52:33.698189+0000","last_clean":"2024-09-11T05:52:33.698189+0000","last_became_active":"2024-09-11T05:52:14.614711+0000","last_became_peered":"2024-09-11T05:52:14.614711+0000","last_unstale":"2024-09-11T05:52:33.698189+0000","last_undegraded":"2024-09-11T05:52:33.698189+0000","last_fullsized":"2024-09-11T05:52:33.698189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:51.273072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":332,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.709244+0000","last_change":"2024-09-11T05:52:23.538843+0000","last_active":"2024-09-11T05:52:33.709244+0000","last_peered":"2024-09-11T05:52:33.709244+0000","last_clean":"2024-09-11T05:52:33.709244+0000","last_became_active":"2024-09-11T05:52:23.537794+0000","last_became_peered":"2024-09-11T05:52:23.537794+0000","last_unstale":"2024-09-11T05:52:33.709244+0000","last_undegraded":"2024-09-11T05:52:33.709244+0000","last_fullsized":"2024-09-11T05:52:33.709244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:19:11.174894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"95'355","reported_seq":5407,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.764577+0000","last_change":"2024-09-11T05:52:08.528671+0000","last_active":"2024-09-11T05:52:33.764577+0000","last_peered":"2024-09-11T05:52:33.764577+0000","last_clean":"2024-09-11T05:52:33.764577+0000","last_became_active":"2024-09-11T05:52:08.527718+0000","last_became_peered":"2024-09-11T05:52:08.527718+0000","last_unstale":"2024-09-11T05:52:33.764577+0000","last_undegraded":"2024-09-11T05:52:33.764577+0000","last_fullsized":"2024-09-11T05:52:33.764577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:43.686416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070988300000000002,"stat_sum":{"num_bytes":37133581,"num_objects":57,"num_object_clones":21,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":19,"num_read":5011,"num_read_kb":15828,"num_write":450,"num_write_kb":17978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"97'223","reported_seq":2833,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.151607+0000","last_change":"2024-09-11T05:51:07.504956+0000","last_active":"2024-09-11T05:52:34.151607+0000","last_peered":"2024-09-11T05:52:34.151607+0000","last_clean":"2024-09-11T05:52:34.151607+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:52:34.151607+0000","last_undegraded":"2024-09-11T05:52:34.151607+0000","last_fullsized":"2024-09-11T05:52:34.151607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:17:48.683004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010921049999999999,"stat_sum":{"num_bytes":11134655,"num_objects":54,"num_object_clones":17,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":23,"num_read":2519,"num_read_kb":10925,"num_write":198,"num_write_kb":4057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"97'570","reported_seq":5246,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.877401+0000","last_change":"2024-09-11T05:51:12.371355+0000","last_active":"2024-09-11T05:52:34.877401+0000","last_peered":"2024-09-11T05:52:34.877401+0000","last_clean":"2024-09-11T05:52:34.877401+0000","last_became_active":"2024-09-11T05:51:12.370444+0000","last_became_peered":"2024-09-11T05:51:12.370444+0000","last_unstale":"2024-09-11T05:52:34.877401+0000","last_undegraded":"2024-09-11T05:52:34.877401+0000","last_fullsized":"2024-09-11T05:52:34.877401+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:44:38.913209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38457667,"num_objects":52,"num_object_clones":14,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":22,"num_read":7400,"num_read_kb":15209,"num_write":666,"num_write_kb":11321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"96'362","reported_seq":4866,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.136106+0000","last_change":"2024-09-11T05:52:23.535746+0000","last_active":"2024-09-11T05:52:34.136106+0000","last_peered":"2024-09-11T05:52:34.136106+0000","last_clean":"2024-09-11T05:52:34.136106+0000","last_became_active":"2024-09-11T05:52:23.534658+0000","last_became_peered":"2024-09-11T05:52:23.534658+0000","last_unstale":"2024-09-11T05:52:34.136106+0000","last_undegraded":"2024-09-11T05:52:34.136106+0000","last_fullsized":"2024-09-11T05:52:34.136106+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:48:15.843597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048178199999999999,"stat_sum":{"num_bytes":33722386,"num_objects":53,"num_object_clones":16,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":20,"num_read":5296,"num_read_kb":20209,"num_write":418,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":3743621,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"96'312","reported_seq":6833,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.136356+0000","last_change":"2024-09-11T05:52:18.644312+0000","last_active":"2024-09-11T05:52:34.136356+0000","last_peered":"2024-09-11T05:52:34.136356+0000","last_clean":"2024-09-11T05:52:34.136356+0000","last_became_active":"2024-09-11T05:52:18.643189+0000","last_became_peered":"2024-09-11T05:52:18.643189+0000","last_unstale":"2024-09-11T05:52:34.136356+0000","last_undegraded":"2024-09-11T05:52:34.136356+0000","last_fullsized":"2024-09-11T05:52:34.136356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:31:20.081682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060246300000000002,"stat_sum":{"num_bytes":18102108,"num_objects":40,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":13,"num_read":6607,"num_read_kb":12060,"num_write":407,"num_write_kb":8983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"97'399","reported_seq":4373,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.181130+0000","last_change":"2024-09-11T05:52:17.643682+0000","last_active":"2024-09-11T05:52:34.181130+0000","last_peered":"2024-09-11T05:52:34.181130+0000","last_clean":"2024-09-11T05:52:34.181130+0000","last_became_active":"2024-09-11T05:52:17.641884+0000","last_became_peered":"2024-09-11T05:52:17.641884+0000","last_unstale":"2024-09-11T05:52:34.181130+0000","last_undegraded":"2024-09-11T05:52:34.181130+0000","last_fullsized":"2024-09-11T05:52:34.181130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:19:08.062737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024846740000000001,"stat_sum":{"num_bytes":22752799,"num_objects":53,"num_object_clones":16,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":19,"num_read":4647,"num_read_kb":16580,"num_write":556,"num_write_kb":8769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"97'833","reported_seq":11177,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.880398+0000","last_change":"2024-09-11T05:51:07.536837+0000","last_active":"2024-09-11T05:52:34.880398+0000","last_peered":"2024-09-11T05:52:34.880398+0000","last_clean":"2024-09-11T05:52:34.880398+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:52:34.880398+0000","last_undegraded":"2024-09-11T05:52:34.880398+0000","last_fullsized":"2024-09-11T05:52:34.880398+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:13:44.633017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417647,"stat_sum":{"num_bytes":64459971,"num_objects":55,"num_object_clones":14,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":19,"num_read":13105,"num_read_kb":32438,"num_write":1122,"num_write_kb":18328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"97'396","reported_seq":2602,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.877561+0000","last_change":"2024-09-11T05:51:07.537622+0000","last_active":"2024-09-11T05:52:34.877561+0000","last_peered":"2024-09-11T05:52:34.877561+0000","last_clean":"2024-09-11T05:52:34.877561+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:52:34.877561+0000","last_undegraded":"2024-09-11T05:52:34.877561+0000","last_fullsized":"2024-09-11T05:52:34.877561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:02:46.784387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020967680000000002,"stat_sum":{"num_bytes":49624104,"num_objects":54,"num_object_clones":16,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":24,"num_read":3093,"num_read_kb":13696,"num_write":585,"num_write_kb":8465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"97'1002","reported_seq":2958,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.764264+0000","last_change":"2024-09-11T05:52:23.539648+0000","last_active":"2024-09-11T05:52:33.764264+0000","last_peered":"2024-09-11T05:52:33.764264+0000","last_clean":"2024-09-11T05:52:33.764264+0000","last_became_active":"2024-09-11T05:52:23.536753+0000","last_became_peered":"2024-09-11T05:52:23.536753+0000","last_unstale":"2024-09-11T05:52:33.764264+0000","last_undegraded":"2024-09-11T05:52:33.764264+0000","last_fullsized":"2024-09-11T05:52:33.764264+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:55:06.112937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44855866,"num_objects":54,"num_object_clones":19,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":19,"num_read":6212,"num_read_kb":22303,"num_write":1647,"num_write_kb":10778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":7741684,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"97'417","reported_seq":3911,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.869486+0000","last_change":"2024-09-11T05:51:07.536990+0000","last_active":"2024-09-11T05:52:34.869486+0000","last_peered":"2024-09-11T05:52:34.869486+0000","last_clean":"2024-09-11T05:52:34.869486+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:52:34.869486+0000","last_undegraded":"2024-09-11T05:52:34.869486+0000","last_fullsized":"2024-09-11T05:52:34.869486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:22.595428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00164964,"stat_sum":{"num_bytes":24063550,"num_objects":64,"num_object_clones":23,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":23,"num_read":4735,"num_read_kb":10938,"num_write":567,"num_write_kb":5121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"97'595","reported_seq":3036,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:33.709364+0000","last_change":"2024-09-11T05:52:23.540294+0000","last_active":"2024-09-11T05:52:33.709364+0000","last_peered":"2024-09-11T05:52:33.709364+0000","last_clean":"2024-09-11T05:52:33.709364+0000","last_became_active":"2024-09-11T05:52:23.537442+0000","last_became_peered":"2024-09-11T05:52:23.537442+0000","last_unstale":"2024-09-11T05:52:33.709364+0000","last_undegraded":"2024-09-11T05:52:33.709364+0000","last_fullsized":"2024-09-11T05:52:33.709364+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:38:12.960315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018113280000000001,"stat_sum":{"num_bytes":10273402,"num_objects":44,"num_object_clones":15,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":17,"num_read":4555,"num_read_kb":13386,"num_write":830,"num_write_kb":6370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":3111704,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"97'319","reported_seq":12392,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-11T05:52:34.896490+0000","last_change":"2024-09-11T05:51:07.538155+0000","last_active":"2024-09-11T05:52:34.896490+0000","last_peered":"2024-09-11T05:52:34.896490+0000","last_clean":"2024-09-11T05:52:34.896490+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:52:34.896490+0000","last_undegraded":"2024-09-11T05:52:34.896490+0000","last_fullsized":"2024-09-11T05:52:34.896490+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:07.003955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019941949999999998,"stat_sum":{"num_bytes":55046261,"num_objects":65,"num_object_clones":22,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":21,"num_read":12444,"num_read_kb":22097,"num_write":414,"num_write_kb":14947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1036579595,"num_objects":1705,"num_object_clones":533,"num_object_copies":3410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1705,"num_whiteouts":643,"num_read":181694,"num_read_kb":504839,"num_write":18394,"num_write_kb":294162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":87221595,"num_keys_recovered":328,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345792512,"data_stored":438141320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13935,"ondisk_log_size":13935,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542169,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99740,"kb_used_data":50504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94272100,"statfs":{"total":96636764160,"available":96534630400,"internally_reserved":0,"allocated":51716096,"data_stored":63539011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542170,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88184,"kb_used_data":44512,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94283656,"statfs":{"total":96636764160,"available":96546463744,"internally_reserved":0,"allocated":45580288,"data_stored":55993951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":60129542169,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542170,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129260,"kb_used_data":75864,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94242580,"statfs":{"total":96636764160,"available":96504401920,"internally_reserved":0,"allocated":77684736,"data_stored":100479727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86872,"kb_used_data":55684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94284968,"statfs":{"total":96636764160,"available":96547807232,"internally_reserved":0,"allocated":57020416,"data_stored":68593283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52184,"kb_used_data":16836,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319656,"statfs":{"total":96636764160,"available":96583327744,"internally_reserved":0,"allocated":17240064,"data_stored":21761188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73628,"kb_used_data":34116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94298212,"statfs":{"total":96636764160,"available":96561369088,"internally_reserved":0,"allocated":34934784,"data_stored":42038991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":60129542170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37187584,"data_stored":47508326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33783808,"data_stored":41508508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16093184,"data_stored":21230724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":63225856,"data_stored":75408994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76537856,"data_stored":99949263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25153536,"data_stored":35244226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":43835392,"data_stored":54873100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49975296,"data_stored":62418179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:52:37.283 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:37.284 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:37.284 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:37.284 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:37.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:37.350 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:37.350+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:37.451 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:37.447+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:37.468 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:37.468 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:37.468 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:37.468 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:37.468 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:37.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:37.551 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:37.550+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:37.652 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:37.651+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:37.657 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.812401000000001, 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:37.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 15.079784543000001, 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.663 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:52:37.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:52:37.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.925458670999999, 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.670 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:52:37.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:37.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133882 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2924544~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2245344750000005, 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070367317999999998, 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.675 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133882 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1100000000000001e-07 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429510+0000", 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1820000000000001e-06 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429513+0000", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3720000000000001e-06 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429515+0000", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499653+0000", 2024-09-11T05:52:37.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070137156000000006 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499708+0000", 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5744e-05 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499875+0000", 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016692000000000001 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.679 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2245235770000003, 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:52:37.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2245129270000001, 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.684 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.224502288, 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.689 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:52:37.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2244901060000002, 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:52:37.693 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:52:37.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.224479133, 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:52:37.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:37.702 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2244683690000002, 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:52:37.706 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2244542330000003, 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.707 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:52:37.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:52:37.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2244437030000004, 2024-09-11T05:52:37.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:37.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:52:37.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:52:37.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2244320220000002, 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:37.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.224421703, 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:52:37.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:52:37.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133897 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3846144~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:37.725 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2209270380000001, 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070534081999999998, 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133897 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:37.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433116+0000", 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433118+0000", 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4229999999999999e-06 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433124+0000", 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1809999999999998e-06 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433129+0000", 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8709999999999996e-06 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503350+0000", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070221042999999997 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503438+0000", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7557000000000002e-05 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503650+0000", 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000211542 2024-09-11T05:52:37.729 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.220908337, 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.734 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2208963380000002, 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:52:37.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:52:37.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2208799639999999, 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.743 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2208666859999999, 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:52:37.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.2208527919999996, 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:37.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:52:37.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:37.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:37.753 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:37.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:52:37.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:37.755 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:37.751+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:37.772 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:37.772 INFO:teuthology.orchestra.run.smithi165.stdout:5616 read 0x20c1069 thru 0x20c4bf3 (0x3b8b bytes) 2024-09-11T05:52:37.776 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:37.780 INFO:teuthology.orchestra.run.smithi165.stdout:5618 trunc from 0xca0503e to 0xee3c0ae 2024-09-11T05:52:37.794 INFO:teuthology.orchestra.run.smithi165.stdout:5619 punch from 0xd1e3170 to 0xd1e9e06, (0x6c96 bytes) 2024-09-11T05:52:37.794 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:37.794 INFO:teuthology.orchestra.run.smithi165.stdout:5622 read 0xbbf7764 thru 0xbc00c65 (0x9502 bytes) 2024-09-11T05:52:37.794 INFO:teuthology.orchestra.run.smithi165.stdout:5624 write 0xab0d0af thru 0xab11af2 (0x4a44 bytes) 2024-09-11T05:52:37.844 INFO:teuthology.orchestra.run.smithi165.stdout:5625 read 0x73b1637 thru 0x73b6131 (0x4afb bytes) 2024-09-11T05:52:37.844 INFO:teuthology.orchestra.run.smithi165.stdout:5626 write 0x24306dd thru 0x243bfed (0xb911 bytes) 2024-09-11T05:52:37.855 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:37.854+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:37.869 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:37.869 INFO:teuthology.orchestra.run.smithi165.stdout:5629 write 0x3c05011 thru 0x3c1105a (0xc04a bytes) 2024-09-11T05:52:37.909 INFO:teuthology.orchestra.run.smithi165.stdout:5630 trunc from 0xee3c0ae to 0x159cf56 2024-09-11T05:52:37.910 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:37.910 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:37.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:37.956 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:37.955+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:37.983 INFO:teuthology.orchestra.run.smithi165.stdout:5631 writesame 0xa04fa2 thru 0xa0b030 data_size 0x1(0x608f bytes) 2024-09-11T05:52:38.053 INFO:teuthology.orchestra.run.smithi165.stdout:5633 writesame 0x69e8e06 thru 0x69f2758 data_size 0x1(0x9953 bytes) 2024-09-11T05:52:38.053 INFO:teuthology.orchestra.run.smithi165.stdout:5634 read 0x238c37e thru 0x239998d (0xd610 bytes) 2024-09-11T05:52:38.054 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.058 INFO:teuthology.orchestra.run.smithi165.stdout:5636 trunc from 0x69f2759 to 0x380962e 2024-09-11T05:52:38.058 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:38.055+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:38.065 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.065 INFO:teuthology.orchestra.run.smithi165.stdout:5641 read 0xc644d thru 0xc68e0 (0x494 bytes) 2024-09-11T05:52:38.066 INFO:teuthology.orchestra.run.smithi165.stdout:5642 punch from 0x1e21217 to 0x1e2176a, (0x553 bytes) 2024-09-11T05:52:38.066 INFO:teuthology.orchestra.run.smithi165.stdout:5643 read 0x234f489 thru 0x2357e73 (0x89eb bytes) 2024-09-11T05:52:38.093 INFO:teuthology.orchestra.run.smithi165.stdout:5644 write 0xdb62689 thru 0xdb6799f (0x5317 bytes) 2024-09-11T05:52:38.096 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:38.096 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:38.097 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:38.097 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:38.097 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:38.114 INFO:teuthology.orchestra.run.smithi165.stdout:5645 write 0xcf77272 thru 0xcf7c089 (0x4e18 bytes) 2024-09-11T05:52:38.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:38.132 INFO:teuthology.orchestra.run.smithi165.stdout:5646 write 0x3f480c6 thru 0x3f53892 (0xb7cd bytes) 2024-09-11T05:52:38.152 INFO:teuthology.orchestra.run.smithi165.stdout:5647 punch from 0x4f249eb to 0x4f2529d, (0x8b2 bytes) 2024-09-11T05:52:38.152 INFO:teuthology.orchestra.run.smithi165.stdout:5648 punch from 0xc7ee711 to 0xc7f7bb9, (0x94a8 bytes) 2024-09-11T05:52:38.152 INFO:teuthology.orchestra.run.smithi165.stdout:5649 read 0x34dfa09 thru 0x34e5c53 (0x624b bytes) 2024-09-11T05:52:38.152 INFO:teuthology.orchestra.run.smithi165.stdout:5650 punch from 0x8638c86 to 0x863fa44, (0x6dbe bytes) 2024-09-11T05:52:38.152 INFO:teuthology.orchestra.run.smithi165.stdout:5651 write 0x4f03191 thru 0x4f0e01a (0xae8a bytes) 2024-09-11T05:52:38.157 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:38.155+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:38.166 INFO:teuthology.orchestra.run.smithi165.stdout:5652 write 0x35485f4 thru 0x3551f42 (0x994f bytes) 2024-09-11T05:52:38.180 INFO:teuthology.orchestra.run.smithi165.stdout:5653 read 0xb373594 thru 0xb37cdd5 (0x9842 bytes) 2024-09-11T05:52:38.180 INFO:teuthology.orchestra.run.smithi165.stdout:5654 read 0x150966d thru 0x150c7a6 (0x313a bytes) 2024-09-11T05:52:38.197 INFO:teuthology.orchestra.run.smithi165.stdout:5656 trunc from 0xdb679a0 to 0x37fcbc1 2024-09-11T05:52:38.229 INFO:teuthology.orchestra.run.smithi165.stdout:5658 punch from 0xf0234c to 0xf0d432, (0xb0e6 bytes) 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.644973596, 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:52:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:52:38.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:52:38.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.644865692, 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:52:38.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.644807373000001, 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.263 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.644753251000001, 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.268 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:52:38.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.272 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643762992999999, 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:52:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:52:38.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643714844, 2024-09-11T05:52:38.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:52:38.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:52:38.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643682578, 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:38.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643652565, 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:52:38.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643623633000001, 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.291 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:52:38.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.643582227, 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.296 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:52:38.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.300 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639935808000001, 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:52:38.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639853, 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:52:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:52:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639775997999999, 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:52:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:52:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.63970451, 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639648548, 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:52:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:52:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639609302, 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639535953999999, 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:52:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:52:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.639506208, 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:52:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.635618990999999, 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.337 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:52:38.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:52:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:52:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.635564603000001, 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:52:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.345 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:38.347 INFO:teuthology.orchestra.run.smithi165.stdout:5659 writesame 0xecc6186 thru 0xecd4272 data_size 0x1(0xe0ed bytes) 2024-09-11T05:52:38.347 INFO:teuthology.orchestra.run.smithi165.stdout:5662 read 0x3635fa thru 0x371696 (0xe09d bytes) 2024-09-11T05:52:38.347 INFO:teuthology.orchestra.run.smithi165.stdout:5663 read 0x7a01699 thru 0x7a06bd0 (0x5538 bytes) 2024-09-11T05:52:38.347 INFO:teuthology.orchestra.run.smithi165.stdout:5664 trunc from 0xecd4273 to 0x686fc7 2024-09-11T05:52:38.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:38.348 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:38.343+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:38.411 INFO:teuthology.orchestra.run.smithi165.stdout:5665 trunc from 0x686fc7 to 0x11237d1 2024-09-11T05:52:38.439 INFO:teuthology.orchestra.run.smithi165.stdout:5666 write 0x505933b thru 0x506314e (0x9e14 bytes) 2024-09-11T05:52:38.447 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:38.443+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:38.459 INFO:teuthology.orchestra.run.smithi165.stdout:5667 read 0x1763723 thru 0x176453c (0xe1a bytes) 2024-09-11T05:52:38.459 INFO:teuthology.orchestra.run.smithi165.stdout:5668 read 0xfd38dc thru 0xfd7ab1 (0x41d6 bytes) 2024-09-11T05:52:38.460 INFO:teuthology.orchestra.run.smithi165.stdout:5669 trunc from 0x506314f to 0x5690b4a 2024-09-11T05:52:38.483 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:38.483 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:38.483 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:38.489 INFO:teuthology.orchestra.run.smithi165.stdout:5670 writesame 0xd12c41e thru 0xd12db58 data_size 0x1(0x173b bytes) 2024-09-11T05:52:38.489 INFO:teuthology.orchestra.run.smithi165.stdout:5671 writesame 0xf0184e thru 0xf04d98 data_size 0x1(0x354b bytes) 2024-09-11T05:52:38.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:38.508 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.508 INFO:teuthology.orchestra.run.smithi165.stdout:5673 read 0x98dcae2 thru 0x98e7862 (0xad81 bytes) 2024-09-11T05:52:38.509 INFO:teuthology.orchestra.run.smithi165.stdout:5675 writesame 0x815405a thru 0x8158726 data_size 0x1d(0x46cd bytes) 2024-09-11T05:52:38.541 INFO:teuthology.orchestra.run.smithi165.stdout:5676 writesame 0xd160a9d thru 0xd165a31 data_size 0x1(0x4f95 bytes) 2024-09-11T05:52:38.541 INFO:teuthology.orchestra.run.smithi165.stdout:5677 writesame 0xae97a73 thru 0xaea78eb data_size 0x81(0xfe79 bytes) 2024-09-11T05:52:38.550 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:38.550+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:38.567 INFO:teuthology.orchestra.run.smithi165.stdout:5678 read 0xb6e83ff thru 0xb6ee1f6 (0x5df8 bytes) 2024-09-11T05:52:38.567 INFO:teuthology.orchestra.run.smithi165.stdout:5679 punch from 0x789548d to 0x789b60c, (0x617f bytes) 2024-09-11T05:52:38.567 INFO:teuthology.orchestra.run.smithi165.stdout:5681 write 0xce830da thru 0xce8cf94 (0x9ebb bytes) 2024-09-11T05:52:38.591 INFO:teuthology.orchestra.run.smithi165.stdout:5682 read 0xa5019d1 thru 0xa510653 (0xec83 bytes) 2024-09-11T05:52:38.591 INFO:teuthology.orchestra.run.smithi165.stdout:5683 read 0x9e077a3 thru 0x9e0d367 (0x5bc5 bytes) 2024-09-11T05:52:38.591 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.599 INFO:teuthology.orchestra.run.smithi165.stdout:5687 trunc from 0xd165a32 to 0x949eb3c 2024-09-11T05:52:38.621 INFO:teuthology.orchestra.run.smithi165.stdout:5688 read 0x776bc57 thru 0x776e5e2 (0x298c bytes) 2024-09-11T05:52:38.625 INFO:teuthology.orchestra.run.smithi165.stdout:5691 trunc from 0x949eb3c to 0xaff74e6 2024-09-11T05:52:38.631 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:38.631 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:38.631 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:38.631 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:38.631 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:38.637 INFO:teuthology.orchestra.run.smithi165.stdout:5692 read 0x336562d thru 0x336add8 (0x57ac bytes) 2024-09-11T05:52:38.637 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.637 INFO:teuthology.orchestra.run.smithi165.stdout:5696 write 0x829fae9 thru 0x82a4e03 (0x531b bytes) 2024-09-11T05:52:38.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:38.650 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:38.647+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:38.656 INFO:teuthology.orchestra.run.smithi165.stdout:5697 read 0x7ee9cc8 thru 0x7eeb07f (0x13b8 bytes) 2024-09-11T05:52:38.656 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.656 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.657 INFO:teuthology.orchestra.run.smithi165.stdout:5701 read 0x583a655 thru 0x583e59c (0x3f48 bytes) 2024-09-11T05:52:38.657 INFO:teuthology.orchestra.run.smithi165.stdout:5702 write 0x27e4fb1 thru 0x27eb0e5 (0x6135 bytes) 2024-09-11T05:52:38.678 INFO:teuthology.orchestra.run.smithi165.stdout:5703 read 0x65f0cfd thru 0x65f949d (0x87a1 bytes) 2024-09-11T05:52:38.678 INFO:teuthology.orchestra.run.smithi165.stdout:5705 read 0x4788e96 thru 0x4791247 (0x83b2 bytes) 2024-09-11T05:52:38.678 INFO:teuthology.orchestra.run.smithi165.stdout:5706 write 0x7a8aaea thru 0x7a8fdd0 (0x52e7 bytes) 2024-09-11T05:52:38.703 INFO:teuthology.orchestra.run.smithi165.stdout:5708 read 0x7982359 thru 0x798e28f (0xbf37 bytes) 2024-09-11T05:52:38.704 INFO:teuthology.orchestra.run.smithi165.stdout:5709 read 0x55b3f28 thru 0x55c09c5 (0xca9e bytes) 2024-09-11T05:52:38.705 INFO:teuthology.orchestra.run.smithi165.stdout:5711 write 0x73afbca thru 0x73b48a1 (0x4cd8 bytes) 2024-09-11T05:52:38.731 INFO:teuthology.orchestra.run.smithi165.stdout:5715 read 0x2189d4f thru 0x2197710 (0xd9c2 bytes) 2024-09-11T05:52:38.731 INFO:teuthology.orchestra.run.smithi165.stdout:5716 write 0x398e99c thru 0x3998a8a (0xa0ef bytes) 2024-09-11T05:52:38.740 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:52:38.750 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:38.750+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:38.756 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.756 INFO:teuthology.orchestra.run.smithi165.stdout:5718 write 0x911ed52 thru 0x912af95 (0xc244 bytes) 2024-09-11T05:52:38.786 INFO:teuthology.orchestra.run.smithi165.stdout:5719 writesame 0xdc0b964 thru 0xdc120e7 data_size 0x35(0x6784 bytes) 2024-09-11T05:52:38.793 INFO:teuthology.orchestra.run.smithi165.stdout:5721 write 0xdcbca91 thru 0xdcc4d47 (0x82b7 bytes) 2024-09-11T05:52:38.808 INFO:teuthology.orchestra.run.smithi165.stdout:5722 writesame 0x6197f69 thru 0x61a55c9 data_size 0x1(0xd661 bytes) 2024-09-11T05:52:38.828 INFO:teuthology.orchestra.run.smithi165.stdout:5724 read 0x78fe08 thru 0x7977d1 (0x79ca bytes) 2024-09-11T05:52:38.828 INFO:teuthology.orchestra.run.smithi165.stdout:5728 write 0x323dd39 thru 0x324ba7b (0xdd43 bytes) 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315418508999997, 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:38.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:52:38.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315386334999999, 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:52:38.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:52:38.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315363314999999, 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:52:38.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:52:38.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:38.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315326831, 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:52:38.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.31529956, 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:52:38.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.858 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315276533000002, 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:52:38.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315252817999998, 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:52:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315230003000003, 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:52:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315207043999997, 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:52:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315184119000001, 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:52:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315160405, 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:52:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315133709000001, 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:52:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315107343000001, 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:52:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.315072647999997, 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:52:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.314893359999999, 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:52:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.312242679000001, 2024-09-11T05:52:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:52:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:52:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.312220629000002, 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:52:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.312007502999997, 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:52:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:52:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.311977339999999, 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:52:38.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.311953443, 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:52:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:38.930 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:38.931 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5730 write 0x7e5cacf thru 0x7e62fb0 (0x64e2 bytes) 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5731 read 0x767bc40 thru 0x7684368 (0x8729 bytes) 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5732 write 0xcab5f2e thru 0xcabd1c4 (0x7297 bytes) 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5733 trunc from 0xdcc4d48 to 0x8031751 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5735 read 0x5dfa05a thru 0x5e097b8 (0xf75f bytes) 2024-09-11T05:52:38.932 INFO:teuthology.orchestra.run.smithi165.stdout:5736 trunc from 0x8031751 to 0x651b4ad 2024-09-11T05:52:38.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:38.933 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:38.927+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:38.957 INFO:teuthology.orchestra.run.smithi165.stdout:5738 writesame 0x4b22c27 thru 0x4b2c062 data_size 0x4(0x943c bytes) 2024-09-11T05:52:38.984 INFO:teuthology.orchestra.run.smithi165.stdout:5742 punch from 0x5d80f50 to 0x5d86d63, (0x5e13 bytes) 2024-09-11T05:52:38.984 INFO:teuthology.orchestra.run.smithi165.stdout:5743 write 0xaade38 thru 0xab675d (0x8926 bytes) 2024-09-11T05:52:39.006 INFO:teuthology.orchestra.run.smithi165.stdout:5744 writesame 0x129db55 thru 0x12a041a data_size 0x2(0x28c6 bytes) 2024-09-11T05:52:39.032 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:39.030+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:39.033 INFO:teuthology.orchestra.run.smithi165.stdout:5745 trunc from 0x651b4ad to 0xa97ce64 2024-09-11T05:52:39.043 INFO:teuthology.orchestra.run.smithi165.stdout:5747 read 0xe96e51 thru 0xea6d69 (0xff19 bytes) 2024-09-11T05:52:39.045 INFO:teuthology.orchestra.run.smithi165.stdout:5749 write 0x38a6590 thru 0x38b2574 (0xbfe5 bytes) 2024-09-11T05:52:39.051 INFO:teuthology.orchestra.run.smithi165.stdout:5750 punch from 0x78223b9 to 0x7826007, (0x3c4e bytes) 2024-09-11T05:52:39.051 INFO:teuthology.orchestra.run.smithi165.stdout:5751 punch from 0x3dd267e to 0x3ddbfd8, (0x995a bytes) 2024-09-11T05:52:39.051 INFO:teuthology.orchestra.run.smithi165.stdout:5752 read 0xa2138fe thru 0xa2150a7 (0x17aa bytes) 2024-09-11T05:52:39.051 INFO:teuthology.orchestra.run.smithi165.stdout:5753 write 0x7ebcf14 thru 0x7ec69c4 (0x9ab1 bytes) 2024-09-11T05:52:39.069 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:39.069 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:39.076 INFO:teuthology.orchestra.run.smithi165.stdout:5757 trunc from 0xa97ce64 to 0x78d30a3 2024-09-11T05:52:39.097 INFO:teuthology.orchestra.run.smithi165.stdout:5760 punch from 0x5d0a059 to 0x5d0cf78, (0x2f1f bytes) 2024-09-11T05:52:39.110 INFO:teuthology.orchestra.run.smithi165.stdout:5761 write 0x8221f4c thru 0x8227d42 (0x5df7 bytes) 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157659 3.1e e100/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 100'639, mlcod=100'634)", 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:39.122831+0000", 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0053424129999999998, 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053779179999999998, 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:39.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122831+0000", 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122831+0000", 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122833+0000", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.866e-06 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122847+0000", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.413e-05 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122928+0000", 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0871999999999994e-05 2024-09-11T05:52:39.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.122948+0000", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9749e-05 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.123516+0000", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00056863400000000004 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.123593+0000", 2024-09-11T05:52:39.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6902000000000003e-05 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157660 3.1e e100/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 100'640, mlcod=100'635)", 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:39.124179+0000", 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0039944000000000004, 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041221959999999998, 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:39.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124179+0000", 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124179+0000", 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124181+0000", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.64e-06 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124195+0000", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042999999999999e-05 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124221+0000", 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6265e-05 2024-09-11T05:52:39.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124241+0000", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9822000000000001e-05 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124874+0000", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063371100000000002 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.124938+0000", 2024-09-11T05:52:39.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3362999999999996e-05 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157661 3.1e e100/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 100'641, mlcod=100'635)", 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:39.126101+0000", 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.002072586, 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002261337, 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:39.136 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126101+0000", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126101+0000", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126103+0000", 2024-09-11T05:52:39.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.81e-06 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126117+0000", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4202000000000001e-05 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126143+0000", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6262e-05 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126161+0000", 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8165999999999999e-05 2024-09-11T05:52:39.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126429+0000", 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026803599999999998 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.126482+0000", 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2340999999999999e-05 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157666 3.c e100/84 3:3726a03e:::rbd_data.1124fa04ae41.0000000000000205:head v 100'349, mlcod=100'347)", 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:39.128016+0000", 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00015701499999999999, 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040606399999999998, 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128016+0000", 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128016+0000", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128018+0000", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128033+0000", 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5139e-05 2024-09-11T05:52:39.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128055+0000", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2311999999999999e-05 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128069+0000", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3463e-05 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128144+0000", 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5351000000000004e-05 2024-09-11T05:52:39.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128169+0000", 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5395999999999999e-05 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157667 3.1e e100/23 3:7958adab:::rbd_data.1124fa04ae41.0000000000000206:head v 100'642, mlcod=100'637)", 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:39.128185+0000", 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4294967295.999989, 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029780600000000001, 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:39.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128185+0000", 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128185+0000", 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128186+0000", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128198+0000", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1873e-05 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128212+0000", 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4102e-05 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128221+0000", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4769999999999992e-06 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128327+0000", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106007 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:39.128363+0000", 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6022000000000002e-05 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.146 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:39.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:39.147 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:52:39.147 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 5 2024-09-11T05:52:39.147 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:39.148 INFO:teuthology.orchestra.run.smithi165.stdout:5762 read 0x3d2c012 thru 0x3d3867f (0xc66e bytes) 2024-09-11T05:52:39.148 INFO:teuthology.orchestra.run.smithi165.stdout:5764 punch from 0x78747b to 0x793878, (0xc3fd bytes) 2024-09-11T05:52:39.148 INFO:teuthology.orchestra.run.smithi165.stdout:5765 read 0x5e4a884 thru 0x5e586e8 (0xde65 bytes) 2024-09-11T05:52:39.148 INFO:teuthology.orchestra.run.smithi165.stdout:5766 write 0x728927 thru 0x72b055 (0x272f bytes) 2024-09-11T05:52:39.149 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:39.147+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:39.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:39.151 INFO:teuthology.orchestra.run.smithi165.stdout:5767 read 0x14de99a thru 0x14e7447 (0x8aae bytes) 2024-09-11T05:52:39.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:39.174 INFO:teuthology.orchestra.run.smithi165.stdout:5768 write 0xc503bc1 thru 0xc50c52b (0x896b bytes) 2024-09-11T05:52:39.195 INFO:teuthology.orchestra.run.smithi165.stdout:5769 write 0x36ec8a8 thru 0x36faaa6 (0xe1ff bytes) 2024-09-11T05:52:39.215 INFO:teuthology.orchestra.run.smithi165.stdout:5770 writesame 0x9d98c1d thru 0x9d9ae19 data_size 0xb(0x21fd bytes) 2024-09-11T05:52:39.237 INFO:teuthology.orchestra.run.smithi165.stdout:5771 write 0xc67b353 thru 0xc67cc00 (0x18ae bytes) 2024-09-11T05:52:39.249 INFO:teuthology.orchestra.run.smithi165.stdout:5774 read 0x3e29890 thru 0x3e2fab6 (0x6227 bytes) 2024-09-11T05:52:39.249 INFO:teuthology.orchestra.run.smithi165.stdout:5776 read 0x5c103a7 thru 0x5c1e0fe (0xdd58 bytes) 2024-09-11T05:52:39.249 INFO:teuthology.orchestra.run.smithi165.stdout:5777 read 0x2fa5a67 thru 0x2faefc3 (0x955d bytes) 2024-09-11T05:52:39.249 INFO:teuthology.orchestra.run.smithi165.stdout:5778 read 0x56a8958 thru 0x56b7267 (0xe910 bytes) 2024-09-11T05:52:39.249 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:39.246+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:39.268 INFO:teuthology.orchestra.run.smithi165.stdout:5779 trunc from 0xc67cc01 to 0x956b66 2024-09-11T05:52:39.333 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:39.333 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:39.334 INFO:teuthology.orchestra.run.smithi165.stdout:5781 punch from 0x189857 to 0x1915e6, (0x7d8f bytes) 2024-09-11T05:52:39.334 INFO:teuthology.orchestra.run.smithi165.stdout:5783 read 0x149eb thru 0x22e16 (0xe42c bytes) 2024-09-11T05:52:39.336 INFO:teuthology.orchestra.run.smithi165.stdout:5785 read 0x2e4cee thru 0x2ec71c (0x7a2f bytes) 2024-09-11T05:52:39.339 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:39.340 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:39.340 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:39.340 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:39.340 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:39.350 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:39.347+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:39.352 INFO:teuthology.orchestra.run.smithi165.stdout:5788 write 0x3e541c4 thru 0x3e558db (0x1718 bytes) 2024-09-11T05:52:39.352 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:39.352 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:39.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:39.358 INFO:teuthology.orchestra.run.smithi165.stdout:5790 punch from 0x1d8a5c0 to 0x1d94651, (0xa091 bytes) 2024-09-11T05:52:39.366 INFO:teuthology.orchestra.run.smithi165.stdout:5791 writesame 0x4fae2b1 thru 0x4fb5170 data_size 0x20(0x6ec0 bytes) 2024-09-11T05:52:39.371 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:39.372 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:39.379 INFO:teuthology.orchestra.run.smithi165.stdout:5792 write 0x92c9ff9 thru 0x92d9646 (0xf64e bytes) 2024-09-11T05:52:39.393 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:39.393 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:39.408 INFO:teuthology.orchestra.run.smithi165.stdout:5793 writesame 0xb16b55a thru 0xb16d90b data_size 0x3(0x23b2 bytes) 2024-09-11T05:52:39.408 INFO:teuthology.orchestra.run.smithi165.stdout:5794 read 0x5e1048e thru 0x5e17993 (0x7506 bytes) 2024-09-11T05:52:39.408 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:39.408 INFO:teuthology.orchestra.run.smithi165.stdout:5796 read 0x6dc6b22 thru 0x6dd55a1 (0xea80 bytes) 2024-09-11T05:52:39.408 INFO:teuthology.orchestra.run.smithi165.stdout:5800 writesame 0x8fb1526 thru 0x8fc12c4 data_size 0x1(0xfd9f bytes) 2024-09-11T05:52:39.416 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:39.416 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:39.433 INFO:teuthology.orchestra.run.smithi165.stdout:5801 write 0x8ac6db8 thru 0x8ad42c6 (0xd50f bytes) 2024-09-11T05:52:39.436 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:39.436 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:39.449 INFO:teuthology.orchestra.run.smithi165.stdout:5803 write 0x75066e8 thru 0x75093b3 (0x2ccc bytes) 2024-09-11T05:52:39.451 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:39.450+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:39.458 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:39.458 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:39.467 INFO:teuthology.orchestra.run.smithi165.stdout:5804 trunc from 0xb16d90c to 0x69a313b 2024-09-11T05:52:39.481 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:39.481 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:39.496 INFO:teuthology.orchestra.run.smithi165.stdout:5805 read 0x48157be thru 0x4822a4a (0xd28d bytes) 2024-09-11T05:52:39.496 INFO:teuthology.orchestra.run.smithi165.stdout:5806 write 0x5685d6d thru 0x5688117 (0x23ab bytes) 2024-09-11T05:52:39.512 INFO:teuthology.orchestra.run.smithi165.stdout:5807 read 0x2e07778 thru 0x2e0d202 (0x5a8b bytes) 2024-09-11T05:52:39.512 INFO:teuthology.orchestra.run.smithi165.stdout:5808 read 0x1ecb122 thru 0x1ed673e (0xb61d bytes) 2024-09-11T05:52:39.515 INFO:teuthology.orchestra.run.smithi165.stdout:5809 trunc from 0x69a313b to 0x8747b9a 2024-09-11T05:52:39.521 INFO:teuthology.orchestra.run.smithi165.stdout:5810 read 0x175340f thru 0x175f9e5 (0xc5d7 bytes) 2024-09-11T05:52:39.522 INFO:teuthology.orchestra.run.smithi165.stdout:5812 read 0x185255d thru 0x1859d3e (0x77e2 bytes) 2024-09-11T05:52:39.535 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:39.535 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:39.538 INFO:teuthology.orchestra.run.smithi165.stdout:5813 write 0xecf428f thru 0xecfd14e (0x8ec0 bytes) 2024-09-11T05:52:39.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:39.553 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:39.550+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:39.553 INFO:teuthology.orchestra.run.smithi165.stdout:5814 read 0xe172de thru 0xe22b38 (0xb85b bytes) 2024-09-11T05:52:39.553 INFO:teuthology.orchestra.run.smithi165.stdout:5817 clone 37 order 18 su 2048 sc 12 2024-09-11T05:52:39.653 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:39.650+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:39.718 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:39.718 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:39.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:39.775 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:39.774+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:39.855 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:39.851+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:39.900 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:39.900 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:39.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:39.956 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:39.951+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:40.000 INFO:teuthology.orchestra.run.smithi165.stdout:truncating image image_client.0-clone36 from 0xecfd14f (overlap 0x686fc7) to 0x29c64f 2024-09-11T05:52:40.050 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.051 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.056 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:40.055+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:40.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:40.157 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:40.154+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:40.217 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.218 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:40.276 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:40.275+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:40.359 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:40.355+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:40.362 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.362 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.363 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-09-11T05:52:40.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:40.464 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:40.463+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:40.552 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.552 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.561 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:40.558+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:40.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:40.662 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:40.659+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:40.743 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.743 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.755 INFO:teuthology.orchestra.run.smithi165.stdout:5818 write 0xca19338 thru 0xca1f9a9 (0x6672 bytes) 2024-09-11T05:52:40.762 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:40.759+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:40.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:40.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:52:40.805 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:40.828 INFO:teuthology.orchestra.run.smithi165.stdout:5820 trunc from 0xecfd14f to 0x31dad8 2024-09-11T05:52:40.870 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:40.867+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:40.961 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:40.961 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:40.963 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:40.959+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:40.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:41.065 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:41.062+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:41.139 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:41.139 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:41.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:41.204 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:41.203+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:41.267 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:41.265+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:41.269 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:41.270 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:41.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:41.376 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:41.375+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:41.452 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:41.452 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:41.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:41.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:41.509 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:41.507+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:41.569 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:41.567+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:41.633 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:41.633 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:41.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:41.662 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:41.663 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:41.680 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:41.681 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:41.681 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:41.679+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:41.755 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:52:41.771 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:41.767+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:41.778 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:41.779 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:41.796 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:41.797 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:41.797 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:41.797 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:41.797 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:41.797 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:41.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:41.817 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:41.818 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:41.841 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:41.841 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:41.862 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:41.862 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:41.872 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:41.867+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:41.885 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:41.885 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:41.933 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:41.933 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:41.933 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:41.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:41.973 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:41.969+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:42.075 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:42.071+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.261470930000002, 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 19.528854472999999, 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:52:42.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:52:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:52:42.115 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.374528601000002, 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:52:42.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:52:42.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:52:42.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133882 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2924544~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:42.120 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6736044050000007, 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070367317999999998, 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133882 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:42.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429508+0000", 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1100000000000001e-07 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429510+0000", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1820000000000001e-06 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429513+0000", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3720000000000001e-06 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429515+0000", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499653+0000", 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070137156000000006 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499708+0000", 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5744e-05 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499875+0000", 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016692000000000001 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.124 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735935069999996, 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.125 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735828569999995, 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.129 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:52:42.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735722180000003, 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735600359999996, 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:52:42.138 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735490629999994, 2024-09-11T05:52:42.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:52:42.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735382990000005, 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.147 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:52:42.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735241629999997, 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.151 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735136330000007, 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:42.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:52:42.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:52:42.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6735019520000005, 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:52:42.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6734916329999994, 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133897 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3846144~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:42.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699969679999995, 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070534081999999998, 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133897 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433115+0000", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433116+0000", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433118+0000", 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4229999999999999e-06 2024-09-11T05:52:42.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433124+0000", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1809999999999998e-06 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433129+0000", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8709999999999996e-06 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503350+0000", 2024-09-11T05:52:42.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070221042999999997 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503438+0000", 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7557000000000002e-05 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503650+0000", 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000211542 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699782669999994, 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:52:42.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699662679999996, 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.177 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:52:42.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699498940000002, 2024-09-11T05:52:42.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.185 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699366159999993, 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:52:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:52:42.189 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.6699227220000008, 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:42.190 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:42.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:42.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:42.194 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:42.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:42.196 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:42.191+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:42.284 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:42.284 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:42.284 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:52:42.285 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.526269 2024-09-11T05:52:42.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.5262691368065545 2024-09-11T05:52:42.327 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:42.328 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:42.328 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:42.328 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:42.332 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:42.331+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:42.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:42.397 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:42.393+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:42.464 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:42.465 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:42.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:42.523 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:42.521+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:42.599 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:42.595+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.034696041, 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.644 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.034588137, 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:52:42.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.034529817999999, 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.653 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:52:42.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:52:42.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:52:42.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.034475695999999, 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.033485438, 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.033437289, 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:52:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.033405023, 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:52:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:52:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.03337501, 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:52:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.033346077999999, 2024-09-11T05:52:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.033304672, 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029658252999999, 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:52:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029575445000001, 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:52:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029498443, 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:52:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029426955, 2024-09-11T05:52:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:52:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029370993000001, 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:52:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:52:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029331747000001, 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:52:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029258399, 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:52:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.029228653000001, 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:52:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:52:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.025341436, 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:52:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.025287047999999, 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:42.739 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:42.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:42.741 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:42.739+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:42.767 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.6 primary-affinity to 0.526269 (8344892) 2024-09-11T05:52:42.841 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:42.837+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:42.872 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:42.872 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:42.872 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:42.872 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:42.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:42.974 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:42.973+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:43.042 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:43.039+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:43.051 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:43.052 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:43.052 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:43.052 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:43.052 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:43.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:43.143 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:43.143+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:43.239 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:43.239 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:43.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:43.239 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720750547999998, 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:52:43.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720718374, 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:52:43.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720695354, 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:52:43.249 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:52:43.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720658870000001, 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.254 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:52:43.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:43.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720631599000001, 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:43.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720608572000003, 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:52:43.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:43.267 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720584856999999, 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.268 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:52:43.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720562041999997, 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:43.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720539082999998, 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:52:43.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:52:43.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720516158000002, 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:52:43.282 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:52:43.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720492444000001, 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.287 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:52:43.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720465748000002, 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.291 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:43.295 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720439382000002, 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:52:43.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720404686999998, 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:52:43.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.720225399, 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:52:43.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.717574718000002, 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:52:43.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.717552668000003, 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:52:43.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:52:43.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.717339541999998, 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:43.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.322 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.717309379, 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:52:43.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:52:43.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.717285482000001, 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:43.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:43.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:52:43.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:52:43.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.331 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:43.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:43.333 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:43.329+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:43.434 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:43.431+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:43.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:43.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:43.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:43.534 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:43.531+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:43.635 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:43.631+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:43.646 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:43.646 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:43.646 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:43.646 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:43.646 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:43.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:43.736 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:43.731+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.582241752000002, 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:43.789 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:43.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.283062694000002, 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:43.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:43.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.879437975000002, 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:43.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:43.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.400493161, 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.803 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:52:43.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:52:43.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.399890489000001, 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:52:43.812 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.398943504999998, 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:52:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:52:43.816 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127919 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'699, mlcod=93'692)", 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.398240626, 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079047579999999996, 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387833+0000", 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387849+0000", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6203000000000001e-05 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387878+0000", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9473000000000001e-05 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387899+0000", 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999999e-05 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391149+0000", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032499149999999999 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391208+0000", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8615e-05 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395665+0000", 2024-09-11T05:52:43.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044570870000000002 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395736+0000", 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0499000000000004e-05 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127920 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'700, mlcod=93'692)", 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.388526+0000", 2024-09-11T05:52:43.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.397545464, 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007289617, 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388526+0000", 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388526+0000", 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388527+0000", 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2640000000000001e-06 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388538+0000", 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.096e-05 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388573+0000", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5234999999999997e-05 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.388588+0000", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.392554+0000", 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039656279999999997 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.392615+0000", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0649000000000002e-05 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395746+0000", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031314989999999998 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395816+0000", 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9292e-05 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127921 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'701, mlcod=93'692)", 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.396788235999999, 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079345010000000001, 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:43.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389285+0000", 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389301+0000", 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389327+0000", 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6142000000000001e-05 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389344+0000", 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7193999999999998e-05 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394014+0000", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046697070000000004 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394073+0000", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9321999999999999e-05 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397074+0000", 2024-09-11T05:52:43.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030010430000000001 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397218+0000", 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014360299999999999 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:43.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.396190400999998, 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:52:43.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:52:43.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127969 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'708, mlcod=93'704)", 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.330377191, 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081173600000000005, 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:43.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455696+0000", 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9139999999999998e-06 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455709+0000", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646000000000001e-05 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455742+0000", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2811999999999997e-05 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455774+0000", 2024-09-11T05:52:43.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2892999999999998e-05 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456234+0000", 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045955200000000001 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456269+0000", 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4842000000000003e-05 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463713+0000", 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007443847 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463812+0000", 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8853999999999999e-05 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.172902022999999, 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:52:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.104151785999999, 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:52:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:52:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 16.035242526000001, 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:52:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:52:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3948367079999997, 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:43.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:52:43.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:52:43.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:52:43.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157015 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'419, mlcod=99'413)", 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3938279839999996, 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076103459999999996, 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.857 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392244+0000", 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3500000000000005e-07 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392250+0000", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2650000000000004e-06 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392263+0000", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069e-05 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392269+0000", 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7619999999999998e-06 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392311+0000", 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2072e-05 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392326+0000", 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4218e-05 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399833+0000", 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075078189999999998 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399854+0000", 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305999999999998e-05 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.2919895180000003, 2024-09-11T05:52:43.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:52:43.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:52:43.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.2912807519999996, 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:52:43.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157101 3.2 e99/83 3:44379fcd:::rbd_data.1124fa04ae41.000000000000003c:head v 99'368, mlcod=99'366)", 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.2868263610000001, 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075074139999999996, 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:43.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499246+0000", 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2020000000000001e-06 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499312+0000", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5473000000000001e-05 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499327+0000", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4915e-05 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499336+0000", 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1330000000000005e-06 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499453+0000", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117045 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499492+0000", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8788000000000002e-05 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506706+0000", 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072144399999999999 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506752+0000", 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6418000000000002e-05 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157247 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'504, mlcod=99'500)", 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0937618640000002, 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073307229999999999, 2024-09-11T05:52:43.875 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:43.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692311+0000", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0270000000000001e-06 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692323+0000", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2016e-05 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692340+0000", 2024-09-11T05:52:43.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7348999999999999e-05 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692351+0000", 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1096e-05 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692419+0000", 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8312999999999995e-05 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:43.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692451+0000", 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1578000000000002e-05 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.699602+0000", 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071514279999999996 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.699640+0000", 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7916000000000001e-05 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.879 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:43.880 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:43.880 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:43.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:52:43.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5822 write 0x477f8cc thru 0x4787867 (0x7f9c bytes) 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5823 punch from 0x33dba1f to 0x33eb6a4, (0xfc85 bytes) 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5825 writesame 0x7030101 thru 0x703602e data_size 0x3e(0x5f2e bytes) 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5827 punch from 0x6a2fdfd to 0x6a33508, (0x370b bytes) 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5828 punch from 0x37de967 to 0x37e8f77, (0xa610 bytes) 2024-09-11T05:52:43.883 INFO:teuthology.orchestra.run.smithi165.stdout:5830 trunc from 0x703602f to 0x72353f 2024-09-11T05:52:43.884 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:43.879+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:43.898 INFO:teuthology.orchestra.run.smithi165.stdout:5832 read 0x49503c thru 0x496fe2 (0x1fa7 bytes) 2024-09-11T05:52:43.899 INFO:teuthology.orchestra.run.smithi165.stdout:5833 read 0x418933 thru 0x423e82 (0xb550 bytes) 2024-09-11T05:52:43.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:43.922 INFO:teuthology.orchestra.run.smithi165.stdout:5834 write 0x955ba24 thru 0x955e7f0 (0x2dcd bytes) 2024-09-11T05:52:43.938 INFO:teuthology.orchestra.run.smithi165.stdout:5835 write 0x71712ac thru 0x717c97e (0xb6d3 bytes) 2024-09-11T05:52:43.967 INFO:teuthology.orchestra.run.smithi165.stdout:5837 punch from 0x81653a0 to 0x81704d9, (0xb139 bytes) 2024-09-11T05:52:43.967 INFO:teuthology.orchestra.run.smithi165.stdout:5840 read 0x5a26acb thru 0x5a2e005 (0x753b bytes) 2024-09-11T05:52:43.968 INFO:teuthology.orchestra.run.smithi165.stdout:5842 write 0x3cbc347 thru 0x3cc3231 (0x6eeb bytes) 2024-09-11T05:52:43.983 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:43.979+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:43.993 INFO:teuthology.orchestra.run.smithi165.stdout:5844 read 0x37f27f thru 0x38b747 (0xc4c9 bytes) 2024-09-11T05:52:43.994 INFO:teuthology.orchestra.run.smithi165.stdout:5845 write 0x118f7ba thru 0x1194cbe (0x5505 bytes) 2024-09-11T05:52:44.016 INFO:teuthology.orchestra.run.smithi165.stdout:5847 read 0x5a78af7 thru 0x5a8450c (0xba16 bytes) 2024-09-11T05:52:44.016 INFO:teuthology.orchestra.run.smithi165.stdout:5848 write 0x36fa22 thru 0x37c674 (0xcc53 bytes) 2024-09-11T05:52:44.022 INFO:teuthology.orchestra.run.smithi165.stdout:5851 read 0x528fd09 thru 0x5296975 (0x6c6d bytes) 2024-09-11T05:52:44.022 INFO:teuthology.orchestra.run.smithi165.stdout:5855 read 0x7bb6c7d thru 0x7bb8a3e (0x1dc2 bytes) 2024-09-11T05:52:44.022 INFO:teuthology.orchestra.run.smithi165.stdout:5857 read 0x14a5f5e thru 0x14b3b69 (0xdc0c bytes) 2024-09-11T05:52:44.030 INFO:teuthology.orchestra.run.smithi165.stdout:5858 trunc from 0x955e7f1 to 0x45d33b2 2024-09-11T05:52:44.036 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:44.036 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:44.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.036 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161759 3.11 3:89313170:::rbd_data.11243956a123.00000000000001c0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:52:44.036 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:44.033585+0000", 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0028243909999999999, 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028572039999999999, 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161759 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:44.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033585+0000", 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033585+0000", 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:44.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033586+0000", 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.879999999999999e-07 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033591+0000", 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1030000000000001e-06 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033607+0000", 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5662999999999999e-05 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033617+0000", 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033677+0000", 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0306999999999997e-05 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033787+0000", 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110438 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.033925+0000", 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013721499999999999 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:44.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:44.034879+0000", 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00095406600000000005 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-11T05:52:44.042 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:44.049 INFO:teuthology.orchestra.run.smithi165.stdout:5859 read 0x2adf647 thru 0x2aee6bd (0xf077 bytes) 2024-09-11T05:52:44.049 INFO:teuthology.orchestra.run.smithi165.stdout:5860 punch from 0x2b89f0 to 0x2c0301, (0x7911 bytes) 2024-09-11T05:52:44.049 INFO:teuthology.orchestra.run.smithi165.stdout:5861 read 0x6c925f thru 0x6d369b (0xa43d bytes) 2024-09-11T05:52:44.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:44.071 INFO:teuthology.orchestra.run.smithi165.stdout:5862 writesame 0xee2376d thru 0xee31a73 data_size 0x1(0xe307 bytes) 2024-09-11T05:52:44.071 INFO:teuthology.orchestra.run.smithi165.stdout:5863 write 0x531d553 thru 0x53294d7 (0xbf85 bytes) 2024-09-11T05:52:44.083 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:44.079+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:44.084 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:44.084 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:44.087 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.087 INFO:teuthology.orchestra.run.smithi165.stdout:5865 read 0x98d3661 thru 0x98d91e5 (0x5b85 bytes) 2024-09-11T05:52:44.087 INFO:teuthology.orchestra.run.smithi165.stdout:5866 writesame 0x401da24 thru 0x402b119 data_size 0x5(0xd6f6 bytes) 2024-09-11T05:52:44.106 INFO:teuthology.orchestra.run.smithi165.stdout:5867 write 0x2e17834 thru 0x2e26d01 (0xf4ce bytes) 2024-09-11T05:52:44.108 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:44.108 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:44.127 INFO:teuthology.orchestra.run.smithi165.stdout:5869 punch from 0xc92cf33 to 0xc935b81, (0x8c4e bytes) 2024-09-11T05:52:44.128 INFO:teuthology.orchestra.run.smithi165.stdout:5870 read 0x9ffd404 thru 0xa00bc9e (0xe89b bytes) 2024-09-11T05:52:44.128 INFO:teuthology.orchestra.run.smithi165.stdout:5871 read 0xe634fee thru 0xe6358d3 (0x8e6 bytes) 2024-09-11T05:52:44.128 INFO:teuthology.orchestra.run.smithi165.stdout:5874 writesame 0x1080697 thru 0x108b26c data_size 0x6a(0xabd6 bytes) 2024-09-11T05:52:44.129 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:44.130 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:44.136 INFO:teuthology.orchestra.run.smithi165.stdout:5876 write 0x2bd0faa thru 0x2bd6e24 (0x5e7b bytes) 2024-09-11T05:52:44.149 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:44.150 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:44.157 INFO:teuthology.orchestra.run.smithi165.stdout:5877 write 0x404f2fc thru 0x405e06a (0xed6f bytes) 2024-09-11T05:52:44.165 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.165 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.165 INFO:teuthology.orchestra.run.smithi165.stdout:5881 punch from 0x73e0190 to 0x73eea88, (0xe8f8 bytes) 2024-09-11T05:52:44.165 INFO:teuthology.orchestra.run.smithi165.stdout:5882 write 0x82e85a2 thru 0x82f3f4b (0xb9aa bytes) 2024-09-11T05:52:44.171 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:44.171 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:44.184 INFO:teuthology.orchestra.run.smithi165.stdout:5883 punch from 0x31e339b to 0x31f23f9, (0xf05e bytes) 2024-09-11T05:52:44.184 INFO:teuthology.orchestra.run.smithi165.stdout:5885 write 0x6b924d thru 0x6b9744 (0x4f8 bytes) 2024-09-11T05:52:44.185 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:44.181+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:44.187 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.187 INFO:teuthology.orchestra.run.smithi165.stdout:5888 write 0x89de28e thru 0x89e28be (0x4631 bytes) 2024-09-11T05:52:44.193 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:44.193 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:44.203 INFO:teuthology.orchestra.run.smithi165.stdout:5889 read 0x72a3083 thru 0x72aeb20 (0xba9e bytes) 2024-09-11T05:52:44.203 INFO:teuthology.orchestra.run.smithi165.stdout:5890 writesame 0xa4f771 thru 0xa541ff data_size 0x1(0x4a8f bytes) 2024-09-11T05:52:44.208 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:44.209 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:44.209 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:44.209 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:44.209 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:44.213 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:44.213 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:44.219 INFO:teuthology.orchestra.run.smithi165.stdout:5891 punch from 0xee0cd1f to 0xee19e4b, (0xd12c bytes) 2024-09-11T05:52:44.219 INFO:teuthology.orchestra.run.smithi165.stdout:5894 punch from 0xe896897 to 0xe89ebe0, (0x8349 bytes) 2024-09-11T05:52:44.219 INFO:teuthology.orchestra.run.smithi165.stdout:5896 punch from 0x7014d5 to 0x70d49d, (0xbfc8 bytes) 2024-09-11T05:52:44.220 INFO:teuthology.orchestra.run.smithi165.stdout:5897 punch from 0xe4c1da5 to 0xe4c682c, (0x4a87 bytes) 2024-09-11T05:52:44.220 INFO:teuthology.orchestra.run.smithi165.stdout:5899 write 0xeafed86 thru 0xeb08e54 (0xa0cf bytes) 2024-09-11T05:52:44.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:44.233 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:44.234 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:44.244 INFO:teuthology.orchestra.run.smithi165.stdout:5900 trunc from 0xee31a74 to 0xa31619c 2024-09-11T05:52:44.263 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.263 INFO:teuthology.orchestra.run.smithi165.stdout:5904 punch from 0x42190e5 to 0x42207a1, (0x76bc bytes) 2024-09-11T05:52:44.263 INFO:teuthology.orchestra.run.smithi165.stdout:5905 punch from 0x5ec8814 to 0x5ed1f88, (0x9774 bytes) 2024-09-11T05:52:44.263 INFO:teuthology.orchestra.run.smithi165.stdout:5906 punch from 0x75952bf to 0x75a386a, (0xe5ab bytes) 2024-09-11T05:52:44.275 INFO:teuthology.orchestra.run.smithi165.stdout:5907 write 0xecfd0a7 thru 0xed08884 (0xb7de bytes) 2024-09-11T05:52:44.285 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:44.283+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:44.300 INFO:teuthology.orchestra.run.smithi165.stdout:5908 writesame 0x56d76d4 thru 0x56e6918 data_size 0x1(0xf245 bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5909 punch from 0x92894b7 to 0x9297a7e, (0xe5c7 bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5910 read 0x49bb51e thru 0x49c0985 (0x5468 bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5912 punch from 0x2a2477a to 0x2a286be, (0x3f44 bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5917 punch from 0x90384fd to 0x903f42a, (0x6f2d bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5919 punch from 0x86d7ea1 to 0x86e06b8, (0x8817 bytes) 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.324 INFO:teuthology.orchestra.run.smithi165.stdout:5923 write 0x804cf6c thru 0x805316f (0x6204 bytes) 2024-09-11T05:52:44.343 INFO:teuthology.orchestra.run.smithi165.stdout:5924 write 0xae3b353 thru 0xae49cef (0xe99d bytes) 2024-09-11T05:52:44.360 INFO:teuthology.orchestra.run.smithi165.stdout:5925 writesame 0x20d194f thru 0x20d8a45 data_size 0xb(0x70f7 bytes) 2024-09-11T05:52:44.380 INFO:teuthology.orchestra.run.smithi165.stdout:5926 read 0x5f39778 thru 0x5f47f2e (0xe7b7 bytes) 2024-09-11T05:52:44.380 INFO:teuthology.orchestra.run.smithi165.stdout:5928 punch from 0x2c0ede6 to 0x2c14cc1, (0x5edb bytes) 2024-09-11T05:52:44.380 INFO:teuthology.orchestra.run.smithi165.stdout:5929 writesame 0xd5c42d9 thru 0xd5cf75a data_size 0x5(0xb482 bytes) 2024-09-11T05:52:44.386 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:44.383+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:44.399 INFO:teuthology.orchestra.run.smithi165.stdout:5930 writesame 0x55e236f thru 0x55ef087 data_size 0x1(0xcd19 bytes) 2024-09-11T05:52:44.418 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:44.419 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:44.420 INFO:teuthology.orchestra.run.smithi165.stdout:5931 trunc from 0xed08885 to 0xbab2e85 2024-09-11T05:52:44.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:44.451 INFO:teuthology.orchestra.run.smithi165.stdout:5932 write 0x100685c thru 0x1014d07 (0xe4ac bytes) 2024-09-11T05:52:44.455 INFO:teuthology.orchestra.run.smithi165.stdout:5934 writesame 0x916e756 thru 0x91765c3 data_size 0x1(0x7e6e bytes) 2024-09-11T05:52:44.477 INFO:teuthology.orchestra.run.smithi165.stdout:5935 write 0x698566e thru 0x699193a (0xc2cd bytes) 2024-09-11T05:52:44.487 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:44.483+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:44.502 INFO:teuthology.orchestra.run.smithi165.stdout:5936 trunc from 0xbab2e85 to 0xe212b2c 2024-09-11T05:52:44.517 INFO:teuthology.orchestra.run.smithi165.stdout:5937 writesame 0xeb633fa thru 0xeb6d782 data_size 0x5(0xa389 bytes) 2024-09-11T05:52:44.524 INFO:teuthology.orchestra.run.smithi165.stdout:5938 trunc from 0xeb6d783 to 0xb45b85d 2024-09-11T05:52:44.530 INFO:teuthology.orchestra.run.smithi165.stdout:5939 punch from 0xe99889 to 0xea7950, (0xe0c7 bytes) 2024-09-11T05:52:44.530 INFO:teuthology.orchestra.run.smithi165.stdout:5940 write 0xb0bc7b9 thru 0xb0c6e1e (0xa666 bytes) 2024-09-11T05:52:44.539 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.539 INFO:teuthology.orchestra.run.smithi165.stdout:5943 read 0x17a90a1 thru 0x17b43c5 (0xb325 bytes) 2024-09-11T05:52:44.539 INFO:teuthology.orchestra.run.smithi165.stdout:5944 write 0x6bf4786 thru 0x6bfe694 (0x9f0f bytes) 2024-09-11T05:52:44.547 INFO:teuthology.orchestra.run.smithi165.stdout:5945 write 0x346f209 thru 0x3470888 (0x1680 bytes) 2024-09-11T05:52:44.559 INFO:teuthology.orchestra.run.smithi165.stdout:5946 write 0xebe4b5 thru 0xece3ad (0xfef9 bytes) 2024-09-11T05:52:44.582 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:44.582 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:44.584 INFO:teuthology.orchestra.run.smithi165.stdout:5948 write 0x805a176 thru 0x8062f13 (0x8d9e bytes) 2024-09-11T05:52:44.588 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:44.583+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:44.593 INFO:teuthology.orchestra.run.smithi165.stdout:5949 write 0x9aa9f49 thru 0x9ab86c0 (0xe778 bytes) 2024-09-11T05:52:44.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:44.617 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.617 INFO:teuthology.orchestra.run.smithi165.stdout:5953 trunc from 0xb45b85d to 0xb6e2bc2 2024-09-11T05:52:44.624 INFO:teuthology.orchestra.run.smithi165.stdout:5955 read 0x2c2ff17 thru 0x2c37153 (0x723d bytes) 2024-09-11T05:52:44.625 INFO:teuthology.orchestra.run.smithi165.stdout:5956 read 0x49a81d0 thru 0x49abe77 (0x3ca8 bytes) 2024-09-11T05:52:44.625 INFO:teuthology.orchestra.run.smithi165.stdout:5958 punch from 0x78d1f28 to 0x78d77dc, (0x58b4 bytes) 2024-09-11T05:52:44.625 INFO:teuthology.orchestra.run.smithi165.stdout:5959 writesame 0x6cf1ad3 thru 0x6cfbbc6 data_size 0x2(0xa0f4 bytes) 2024-09-11T05:52:44.656 INFO:teuthology.orchestra.run.smithi165.stdout:5960 read 0x3a65278 thru 0x3a6ee6a (0x9bf3 bytes) 2024-09-11T05:52:44.656 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:44.658 INFO:teuthology.orchestra.run.smithi165.stdout:5962 trunc from 0xb6e2bc2 to 0xc683bab 2024-09-11T05:52:44.666 INFO:teuthology.orchestra.run.smithi165.stdout:5963 writesame 0x6d479e6 thru 0x6d484c4 data_size 0xb(0xadf bytes) 2024-09-11T05:52:44.668 INFO:teuthology.orchestra.run.smithi165.stdout:5964 read 0x916bf01 thru 0x9171453 (0x5553 bytes) 2024-09-11T05:52:44.669 INFO:teuthology.orchestra.run.smithi165.stdout:5966 read 0x746c6cc thru 0x7479466 (0xcd9b bytes) 2024-09-11T05:52:44.670 INFO:teuthology.orchestra.run.smithi165.stdout:5967 read 0x40954a4 thru 0x40a2be1 (0xd73e bytes) 2024-09-11T05:52:44.670 INFO:teuthology.orchestra.run.smithi165.stdout:5968 punch from 0x60893 to 0x6e022, (0xd78f bytes) 2024-09-11T05:52:44.681 INFO:teuthology.orchestra.run.smithi165.stdout:5969 write 0xe4d1564 thru 0xe4d92e5 (0x7d82 bytes) 2024-09-11T05:52:44.689 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:44.687+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:44.712 INFO:teuthology.orchestra.run.smithi165.stdout:5970 writesame 0x7a9f69d thru 0x7aa1928 data_size 0x16(0x228c bytes) 2024-09-11T05:52:44.750 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:44.751 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:44.751 INFO:teuthology.orchestra.run.smithi165.stdout:5971 trunc from 0xe4d92e6 to 0x296d32c 2024-09-11T05:52:44.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:44.783 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:52:44.815 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:44.811+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:44.891 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:44.887+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:44.917 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:44.917 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:44.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:44.993 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:44.989+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:45.094 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:45.089+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:45.094 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.094 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:45.195 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:45.193+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:45.231 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.231 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:45.296 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:45.293+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:45.364 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.364 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:45.424 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:45.423+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:45.498 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:45.495+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:45.501 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.501 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:45.608 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:45.607+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:45.690 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.691 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.699 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:45.695+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:45.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:45.799 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:45.795+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:45.831 INFO:teuthology.orchestra.run.smithi165.stdout:5972 read 0x13acfe thru 0x14284e (0x7b51 bytes) 2024-09-11T05:52:45.868 INFO:teuthology.orchestra.run.smithi165.stdout:5973 write 0xb1ffca6 thru 0xb20f5f7 (0xf952 bytes) 2024-09-11T05:52:45.883 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:45.883 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:45.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:45.901 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:45.899+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:45.919 INFO:teuthology.orchestra.run.smithi165.stdout:5974 write 0xecc06e9 thru 0xecc17ee (0x1106 bytes) 2024-09-11T05:52:45.931 INFO:teuthology.orchestra.run.smithi165.stdout:5975 writesame 0x336ee90 thru 0x33704e3 data_size 0x2(0x1654 bytes) 2024-09-11T05:52:45.946 INFO:teuthology.orchestra.run.smithi165.stdout:5976 punch from 0x9d47029 to 0x9d4c510, (0x54e7 bytes) 2024-09-11T05:52:45.946 INFO:teuthology.orchestra.run.smithi165.stdout:5977 writesame 0xc982374 thru 0xc98f19e data_size 0x3(0xce2b bytes) 2024-09-11T05:52:45.983 INFO:teuthology.orchestra.run.smithi165.stdout:5978 write 0x9430e1a thru 0x94329d3 (0x1bba bytes) 2024-09-11T05:52:45.996 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:45.996 INFO:teuthology.orchestra.run.smithi165.stdout:5982 write 0x642d580 thru 0x642f550 (0x1fd1 bytes) 2024-09-11T05:52:46.001 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:45.997+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:46.015 INFO:teuthology.orchestra.run.smithi165.stdout:5983 read 0x32a2375 thru 0x32ab6a2 (0x932e bytes) 2024-09-11T05:52:46.015 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:46.015 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:46.016 INFO:teuthology.orchestra.run.smithi165.stdout:5986 trunc from 0xecc17ef to 0xdaf2b11 2024-09-11T05:52:46.033 INFO:teuthology.orchestra.run.smithi165.stdout:compare and write disabled 2024-09-11T05:52:46.035 INFO:teuthology.orchestra.run.smithi165.stdout:5989 trunc from 0xdaf2b11 to 0xca6881c 2024-09-11T05:52:46.050 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:46.050 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:46.055 INFO:teuthology.orchestra.run.smithi165.stdout:5990 read 0x9200235 thru 0x92027a8 (0x2574 bytes) 2024-09-11T05:52:46.055 INFO:teuthology.orchestra.run.smithi165.stdout:5991 writesame 0x11e4bf8 thru 0x11ee59b data_size 0x2(0x99a4 bytes) 2024-09-11T05:52:46.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:46.069 INFO:teuthology.orchestra.run.smithi165.stdout:5992 write 0x3666811 thru 0x36723ee (0xbbde bytes) 2024-09-11T05:52:46.106 INFO:teuthology.orchestra.run.smithi165.stdout:5993 read 0x151b755 thru 0x1526fe4 (0xb890 bytes) 2024-09-11T05:52:46.107 INFO:teuthology.orchestra.run.smithi165.stdout:5994 punch from 0xa5e5d3c to 0xa5f1954, (0xbc18 bytes) 2024-09-11T05:52:46.107 INFO:teuthology.orchestra.run.smithi165.stdout:5997 punch from 0xc85f2ca to 0xc86d174, (0xdeaa bytes) 2024-09-11T05:52:46.107 INFO:teuthology.orchestra.run.smithi165.stdout:6000 read 0x8ede4c thru 0x8fa044 (0xc1f9 bytes) 2024-09-11T05:52:46.107 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:46.103+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:46.196 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:46.196 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:46.203 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:46.201+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:46.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:46.216 INFO:teuthology.orchestra.run.smithi165.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-09-11T05:52:46.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:46.308 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:46.307+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:46.364 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:46.364 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:46.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:46.424 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:46.423+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:46.442 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:46.442 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:46.460 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:46.460 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:46.478 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:46.478 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:46.499 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:46.499 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:46.504 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:46.501+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:46.528 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:46.528 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:164530 3.f e107/23 3:f37042a2:::rbd_object_map.11243956a123:head v 107'865, mlcod=107'863)", 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:46.534134+0000", 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0013647500000000001, 2024-09-11T05:52:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001416633, 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534134+0000", 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534134+0000", 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534135+0000", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.167e-06 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534143+0000", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5590000000000006e-06 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534164+0000", 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0445000000000002e-05 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534177+0000", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3584000000000001e-05 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534282+0000", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104181 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:46.534323+0000", 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1321e-05 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:52:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-11T05:52:46.539 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:46.548 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:46.549 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:46.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:46.566 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:46.566 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:46.585 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:46.585 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:46.649 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:46.647+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:46.705 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:46.701+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:46.719 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:46.719 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:46.719 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:46.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:46.806 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:46.803+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:46.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:52:46.906 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:46.903+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:46.920 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:46.920 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:46.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.312222507000001, 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:52:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:52:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.312114603000001, 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:52:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.312056284000001, 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:52:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:52:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.312002161999999, 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:52:46.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:52:46.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.311011904000001, 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.310963755, 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.944 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.948 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.310931489000001, 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:52:46.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:52:46.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.310901476000002, 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:52:46.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:52:46.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.310872543999999, 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.959 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.310831138000001, 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:52:46.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:52:46.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.307184718999999, 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:46.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:52:46.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:52:46.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.307101911, 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:46.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:52:46.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.307024908999999, 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:52:46.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.306953420999999, 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:46.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.991 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.306897458999998, 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.306858213000002, 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:46.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:46.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:46.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:46.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.306784865000001, 2024-09-11T05:52:47.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:47.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.306755119000002, 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.008 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.302867901999999, 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:52:47.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:52:47.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:52:47.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:52:47.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.302813514, 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.013 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.017 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:47.019 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:47.015+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:47.120 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:47.117+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:47.149 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:47.150 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:47.150 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:47.150 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:47.251 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:47.249+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:47.321 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:47.317+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:47.328 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:47.328 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:47.328 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:47.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:47.329 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:47.430 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:47.429+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:47.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992618794000002, 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:52:47.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992586619999997, 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:52:47.514 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:52:47.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992563599999997, 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:52:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:52:47.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:52:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992527115999998, 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.523 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992499844999998, 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.527 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:47.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:52:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992476818, 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:52:47.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.535 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992453103000003, 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.536 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992430288000001, 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:52:47.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:52:47.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992407329000002, 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:52:47.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992384403999999, 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:52:47.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:52:47.553 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992360689999998, 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.554 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:52:47.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992333993999999, 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:52:47.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:52:47.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.562 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992307627999999, 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.563 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:52:47.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:52:47.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:52:47.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992272933000002, 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.567 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:47.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.571 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.992093644999997, 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.989442963999998, 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:52:47.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:52:47.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:52:47.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:52:47.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.989420914, 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.580 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:52:47.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.989207788000002, 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:52:47.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.989177624999996, 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.590 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:52:47.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.989153727999998, 2024-09-11T05:52:47.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:47.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:52:47.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:47.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:47.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:47.599 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:47.601 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:47.597+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:47.701 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:47.697+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:47.772 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:47.772 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:47.773 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:47.794 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:47.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:47.794 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-11T05:52:47.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-11T05:52:47.825 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:52:47.825 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:47.823+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:47.903 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:47.899+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:47.962 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:47.962 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:47.962 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:47.962 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:47.962 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:47.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:48.022 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:48.021+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:48.106 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:48.103+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.900972972000005, 2024-09-11T05:52:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.601793913999998, 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.198169194999998, 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.719224381, 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:52:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.718621709000001, 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:52:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:52:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:52:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.717674724999998, 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:52:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:52:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127919 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'699, mlcod=93'692)", 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.716971846, 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079047579999999996, 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387833+0000", 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387849+0000", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6203000000000001e-05 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387878+0000", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9473000000000001e-05 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387899+0000", 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999999e-05 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391149+0000", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032499149999999999 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391208+0000", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8615e-05 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395665+0000", 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044570870000000002 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395736+0000", 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0499000000000004e-05 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127921 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'701, mlcod=93'692)", 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.715519455999999, 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079345010000000001, 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389285+0000", 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-09-11T05:52:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389301+0000", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389327+0000", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6142000000000001e-05 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389344+0000", 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7193999999999998e-05 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394014+0000", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046697070000000004 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394073+0000", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9321999999999999e-05 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397074+0000", 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030010430000000001 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397218+0000", 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014360299999999999 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.714921620999998, 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:52:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:52:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:52:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127969 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'708, mlcod=93'704)", 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.649108411, 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081173600000000005, 2024-09-11T05:52:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455696+0000", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9139999999999998e-06 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455709+0000", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646000000000001e-05 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455742+0000", 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2811999999999997e-05 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455774+0000", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2892999999999998e-05 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456234+0000", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045955200000000001 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456269+0000", 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4842000000000003e-05 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463713+0000", 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007443847 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463812+0000", 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8853999999999999e-05 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:52:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.491633242999999, 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.422883005999999, 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:52:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.353973746000001, 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:52:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7135679279999998, 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157015 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'419, mlcod=99'413)", 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7125592039999997, 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076103459999999996, 2024-09-11T05:52:48.167 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392244+0000", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3500000000000005e-07 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392250+0000", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2650000000000004e-06 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392263+0000", 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069e-05 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392269+0000", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7619999999999998e-06 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392311+0000", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2072e-05 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392326+0000", 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4218e-05 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399833+0000", 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075078189999999998 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399854+0000", 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305999999999998e-05 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:52:48.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.6107207379999995, 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:52:48.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.6100119720000006, 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:52:48.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:52:48.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157101 3.2 e99/83 3:44379fcd:::rbd_data.1124fa04ae41.000000000000003c:head v 99'368, mlcod=99'366)", 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:48.180 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.6055575809999993, 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075074139999999996, 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:48.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499246+0000", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2020000000000001e-06 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499312+0000", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5473000000000001e-05 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499327+0000", 2024-09-11T05:52:48.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4915e-05 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499336+0000", 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1330000000000005e-06 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499453+0000", 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117045 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499492+0000", 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8788000000000002e-05 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506706+0000", 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072144399999999999 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506752+0000", 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6418000000000002e-05 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157247 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'504, mlcod=99'500)", 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4124930839999994, 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073307229999999999, 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:48.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692310+0000", 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692311+0000", 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0270000000000001e-06 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692323+0000", 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2016e-05 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692340+0000", 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7348999999999999e-05 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692351+0000", 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1096e-05 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692419+0000", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8312999999999995e-05 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.692451+0000", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1578000000000002e-05 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.699602+0000", 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071514279999999996 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.699640+0000", 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7916000000000001e-05 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.5734294630000001, 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:52:48.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:52:48.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:52:48.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.196 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.196 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.196 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:48.196 INFO:tasks.ceph.mon.a.smithi165.stderr:2024-09-11T05:52:48.139+0000 7f2b6abd3640 -1 mon.a@0(leader).osd e109 definitely_dead 0 2024-09-11T05:52:48.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:48.207 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:48.203+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:48.308 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:48.303+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:48.367 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:48.368 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:48.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:48.408 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:48.407+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:48.499 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:48.499 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:48.509 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:48.507+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:48.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:48.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:48.639 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:48.639 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.148812340999999, 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.640 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.145520101000001, 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.145484183000001, 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.648 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:48.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:52:48.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:52:48.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.652 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.140753732, 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.140678779, 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:52:48.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:52:48.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.14061255, 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:52:48.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.140546000000001, 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.137422764, 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.137331875999999, 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:52:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:52:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:52:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.137256853, 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.137185447, 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.133054753, 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.687 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:52:48.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:48.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.132991476999999, 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:52:48.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.132918687, 2024-09-11T05:52:48.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:52:48.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:52:48.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.132845881, 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:52:48.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:52:48.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:52:48.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.132785748, 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.705 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:48.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.13269468, 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:48.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:52:48.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:52:48.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.129370028, 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.714 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:52:48.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.129259528, 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.718 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:48.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.129181805, 2024-09-11T05:52:48.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:52:48.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:52:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:48.727 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:48.727 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:48.647+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:48.728 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:48.725+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:48.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:48.757 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:48.757 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:48.774 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:48.774 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:48.791 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:48.791 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:48.807 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:48.808 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:48.828 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:48.828 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:48.828 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:48.827+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:48.842 INFO:teuthology.orchestra.run.smithi165.stderr:marked in osd.0. 2024-09-11T05:52:48.845 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:48.846 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:48.861 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:48.861 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:48.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:48.873 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-11T05:52:48.885 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:48.886 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:48.906 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:48.906 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:48.930 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:48.925+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:48.998 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:48.998 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:49.056 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:49.055+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:49.127 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.127 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.131 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:49.129+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:49.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:49.232 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:49.229+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:49.306 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.306 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:49.333 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:49.329+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:49.434 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:49.429+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:49.441 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.441 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:49.535 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:49.531+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:49.575 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.575 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:49.680 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:49.679+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:49.737 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:49.733+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:49.756 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.756 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:49.838 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:49.833+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:49.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:52:49.938 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:49.935+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:49.941 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:49.942 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:49.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:50.039 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:50.037+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:50.120 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.120 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:50.181 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:50.179+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:50.240 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:50.237+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:50.252 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.252 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:50.341 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:50.337+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:50.420 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.420 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:50.477 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:50.475+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:50.543 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:50.539+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:50.557 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.557 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:50.645 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:50.643+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:50.704 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.704 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:50.746 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:50.743+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:50.837 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:50.837 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:50.847 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:52:50.847 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:50.845+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:50.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:50.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:50.973 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:50.971+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:51.017 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:51.017 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:51.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:51.072 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:51.072 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:51.078 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:51.077+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:51.088 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:51.089 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:51.105 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:51.105 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:51.126 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:51.126 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:51.150 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:51.147+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:51.153 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:51.154 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:51.159 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:51.159 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:51.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:51.180 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:51.181 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:51.203 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:51.204 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:51.228 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:51.228 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:51.263 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:51.261+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:51.344 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:51.345 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:51.351 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:51.349+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:51.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:51.452 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:51.447+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:51.536 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015591450999999, 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:52:51.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015559277000001, 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.541 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:52:51.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:51.545 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015536257000001, 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:51.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:52:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:52:51.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015499773000002, 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:51.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:52:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015472502000001, 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:52:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:52:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015449474999997, 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:52:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015425759999999, 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015402944999998, 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:51.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:52:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015379985999999, 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.573 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:52:51.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015357061000003, 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:52:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:52:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:52:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:52:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015333347000002, 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:52:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:52:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015306651000003, 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:52:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:52:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015280285000003, 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:52:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015245589999999, 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.594 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:52:51.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.015066302000001, 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.599 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:52:51.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:51.603 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.012415621000002, 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:51.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:52:51.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:52:51.607 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.012393570999997, 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.608 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.012180444999998, 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:52:51.612 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:52:51.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.616 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.012150282, 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.617 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:52:51.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.012126385000002, 2024-09-11T05:52:51.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:52:51.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:52:51.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:51.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:51.626 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:51.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:51.627 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:51.623+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:51.728 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:51.725+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:51.752 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:51.752 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:51.753 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:51.753 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:51.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:51.858 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:51.857+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:51.930 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:51.925+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:51.941 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:51.941 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:51.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:52.046 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:52.045+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.916594570000001, 2024-09-11T05:52:52.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.617415511999994, 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:52.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:52.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.213790793000001, 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:52.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:52.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:52.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.734845978999999, 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:52:52.136 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.734243307, 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:52.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.733296323000001, 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:52:52.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127919 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'699, mlcod=93'692)", 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.732593443999999, 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079047579999999996, 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.149 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387833+0000", 2024-09-11T05:52:52.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387849+0000", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6203000000000001e-05 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387878+0000", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9473000000000001e-05 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387899+0000", 2024-09-11T05:52:52.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999999e-05 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391149+0000", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032499149999999999 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391208+0000", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8615e-05 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395665+0000", 2024-09-11T05:52:52.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044570870000000002 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395736+0000", 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0499000000000004e-05 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127921 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'701, mlcod=93'692)", 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.731141053999998, 2024-09-11T05:52:52.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079345010000000001, 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389285+0000", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389301+0000", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389327+0000", 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6142000000000001e-05 2024-09-11T05:52:52.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389344+0000", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7193999999999998e-05 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394014+0000", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046697070000000004 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394073+0000", 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9321999999999999e-05 2024-09-11T05:52:52.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397074+0000", 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030010430000000001 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397218+0000", 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014360299999999999 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.157 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.730543219000001, 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:52.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:52:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127969 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'708, mlcod=93'704)", 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.664730008999999, 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081173600000000005, 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455696+0000", 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9139999999999998e-06 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455709+0000", 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646000000000001e-05 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455742+0000", 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2811999999999997e-05 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455774+0000", 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2892999999999998e-05 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456234+0000", 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045955200000000001 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456269+0000", 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4842000000000003e-05 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463713+0000", 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007443847 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463812+0000", 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8853999999999999e-05 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.507254840999998, 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:52:52.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.170 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.438504603999998, 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:52:52.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:52:52.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.369595344, 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.175 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:52:52.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:52:52.179 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.729189526000001, 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:52:52.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:52:52.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.183 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157015 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'419, mlcod=99'413)", 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.728180802000001, 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076103459999999996, 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392244+0000", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3500000000000005e-07 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392250+0000", 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2650000000000004e-06 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392263+0000", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069e-05 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392269+0000", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7619999999999998e-06 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392311+0000", 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2072e-05 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392326+0000", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4218e-05 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399833+0000", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075078189999999998 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399854+0000", 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305999999999998e-05 2024-09-11T05:52:52.187 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.626342336, 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:52:52.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.62563357, 2024-09-11T05:52:52.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:52:52.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.196 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157101 3.2 e99/83 3:44379fcd:::rbd_data.1124fa04ae41.000000000000003c:head v 99'368, mlcod=99'366)", 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.621179179, 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075074139999999996, 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499246+0000", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2020000000000001e-06 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499312+0000", 2024-09-11T05:52:52.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5473000000000001e-05 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499327+0000", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4915e-05 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499336+0000", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1330000000000005e-06 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499453+0000", 2024-09-11T05:52:52.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117045 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499492+0000", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8788000000000002e-05 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506706+0000", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072144399999999999 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506752+0000", 2024-09-11T05:52:52.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6418000000000002e-05 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.5890510609999993, 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.201 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:52:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:52:52.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:3715 3.c e110/84 3:30a393f2:::rbd_data.11242e6dd3fe.0000000000000054:25 v 110'398, mlcod=109'395)", 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:52.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.265569653, 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077768000000000004, 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:52.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854856+0000", 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854867+0000", 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1535999999999999e-05 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854893+0000", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5969000000000002e-05 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854914+0000", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645999999999999e-05 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855685+0000", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077148799999999997 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855743+0000", 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7417999999999998e-05 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862514+0000", 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006771628 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862631+0000", 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117105 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.212 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.212 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:52.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:52.214 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:52.209+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:52.314 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:52.313+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:52.382 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:52.382 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:52.382 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:52.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:52.438 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:52.437+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:52.516 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:52.511+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:52.563 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:52.563 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:52.563 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:52.564 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:52.564 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:52.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:52.616 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:52.615+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:52.705 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.214515279, 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:52:52.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:52:52.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.211223039, 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:52:52.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.715 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.211187121, 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:52:52.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.20645667, 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:52:52.720 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.206381716999999, 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.725 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:52.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.729 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.206315488, 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.730 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:52:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.206248938, 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:52:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:52:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.203125701999999, 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:52:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:52:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.203034814, 2024-09-11T05:52:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:52:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:52:52.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.747 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.202959791, 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:52.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.202888385, 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:52:52.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198757691000001, 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.760 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198694415, 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:52:52.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:52:52.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198621624999999, 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.765 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:52:52.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:52:52.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:52:52.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198548818999999, 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:52.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.773 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198488685999999, 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:52:52.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.198397618, 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:52:52.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.195072966, 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.782 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:52.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:52:52.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:52:52.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:52:52.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.194962466, 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.194884742999999, 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:52:52.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:52:52.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:52.796 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:52.797 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:52.793+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:52.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:52.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:52:52.932 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:52.931+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:165449 3.b e114/111 3:d4a2b267:::rbd_header.1124fd5e4ce2:head v 114'456, mlcod=28'1)", 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:52.949576+0000", 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00056161099999999995, 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057842799999999997, 2024-09-11T05:52:52.950 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949576+0000", 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:52.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949576+0000", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949577+0000", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1699999999999997e-07 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949585+0000", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9500000000000001e-06 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:52.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949601+0000", 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5824e-05 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949610+0000", 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0620000000000007e-06 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949669+0000", 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9064999999999998e-05 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:52.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.949695+0000", 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6361000000000001e-05 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 1 2024-09-11T05:52:52.954 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:52.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:53.013 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:53.011+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:53.100 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:53.097+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:53.138 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:53.138 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:53.138 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:53.139 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:53.139 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:53.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:53.201 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:53.197+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:53.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:53.277 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:53.277 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:53.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:53.277 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473258147999999, 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.278 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:53.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:53.282 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473227889, 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:53.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:53.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473182604000002, 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:53.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473149737999996, 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473121683000002, 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.296 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:53.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.301 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:53.302 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473094617000001, 2024-09-11T05:52:53.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:53.302 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.302 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.303 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:53.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473044616000003, 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.308 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:53.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:53.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.473012132999997, 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:53.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:53.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:53.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:53.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472983663999997, 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:53.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:53.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472954784999999, 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:53.322 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:53.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472897967000002, 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.327 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:53.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:53.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:53.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472866934999999, 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:53.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.335 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472838916000001, 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.336 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472811987, 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.340 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:53.344 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472785420999998, 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:53.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:53.348 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472748209999999, 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:53.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472719656999999, 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.353 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:53.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.357 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472671196, 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:53.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:53.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:53.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.362 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.472632580999999, 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:53.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:165419 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.309177877, 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94067595999999998, 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:53.368 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 165419 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:53.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966867+0000", 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3560000000000001e-06 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966882+0000", 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966920+0000", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7945999999999999e-05 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966943+0000", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2932e-05 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967101+0000", 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157646 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967123+0000", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2478999999999999e-05 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907405+0000", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94028174200000003 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907540+0000", 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134542 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:53.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:53.373 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:53.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:53.374 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:53.371+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:53.393 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:53.393 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:53.411 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:53.411 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:53.431 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:53.431 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:53.448 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:53.448 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:53.471 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:53.471 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:53.474 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:53.473+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:53.494 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:53.494 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:53.514 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:53.514 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:53.516 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:53.516 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:53.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:53.538 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:53.539 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:53.600 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:53.599+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:53.656 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:53.657 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:53.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:53.675 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:53.673+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:53.775 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:53.773+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:53.842 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:53.842 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:53.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:53.874 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:53.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:53.874 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:52:53.874 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-11T05:52:53.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-11T05:52:53.902 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:52:53.903 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:53.899+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:53.977 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:53.973+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:54.007 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.007 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:54.077 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:54.075+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:54.163 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.163 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.178 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:54.175+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:54.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:54.279 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:54.277+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:54.360 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.360 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:54.380 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:54.377+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:54.480 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:54.477+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:54.501 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.501 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:54.581 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:54.577+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:54.635 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.635 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:54.682 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:54.677+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:54.778 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.778 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.783 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:54.779+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:54.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:54.884 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:54.881+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:54.907 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T05:52:54.933 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:54.933 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:54.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:54.994 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:54.993+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:55.085 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:55.083+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:55.117 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:55.117 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:55.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:55.186 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:55.181+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:55.254 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:55.254 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:55.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:55.311 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:55.309+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:55.387 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:55.385+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:55.437 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:55.437 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:55.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:55.499 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:55.497+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:55.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:52:55.605 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:55.603+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:55.637 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:55.637 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:55.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:55.689 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:55.687+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:55.722 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:55.723 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:55.742 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:55.742 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:55.766 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:55.766 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:55.775 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:55.776 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:55.784 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:55.784 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:55.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:52:55.791 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:55.787+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:55.803 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:55.804 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:55.823 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:55.823 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:55.841 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:55.842 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:55.866 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:55.867 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:55.891 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:55.889+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:55.914 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:55.915 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:55.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:52:55.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:52:56.018 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:56.017+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:56.093 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:56.089+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:56.096 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:56.096 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:56.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:52:56.194 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:56.191+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.281 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.074503833999998, 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:52:56.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.285 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.775324776000005, 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:56.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:52:56.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:52:56.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:52:56.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.371700056999998, 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.892755243, 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:52:56.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.892152571, 2024-09-11T05:52:56.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:52:56.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.891205587000002, 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.306 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127919 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'699, mlcod=93'692)", 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.890502708, 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079047579999999996, 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:56.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387833+0000", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387849+0000", 2024-09-11T05:52:56.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6203000000000001e-05 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387878+0000", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9473000000000001e-05 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387899+0000", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999999e-05 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391149+0000", 2024-09-11T05:52:56.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032499149999999999 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391208+0000", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8615e-05 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395665+0000", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044570870000000002 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395736+0000", 2024-09-11T05:52:56.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0499000000000004e-05 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127921 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'701, mlcod=93'692)", 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.889050317999999, 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079345010000000001, 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389285+0000", 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389301+0000", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389327+0000", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6142000000000001e-05 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389344+0000", 2024-09-11T05:52:56.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7193999999999998e-05 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394014+0000", 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046697070000000004 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394073+0000", 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9321999999999999e-05 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397074+0000", 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030010430000000001 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397218+0000", 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014360299999999999 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:56.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.888452483000002, 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:52:56.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:52:56.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.319 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127969 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'708, mlcod=93'704)", 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.822639273, 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081173600000000005, 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455696+0000", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9139999999999998e-06 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455709+0000", 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646000000000001e-05 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455742+0000", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2811999999999997e-05 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455774+0000", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2892999999999998e-05 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456234+0000", 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045955200000000001 2024-09-11T05:52:56.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456269+0000", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4842000000000003e-05 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463713+0000", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007443847 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463812+0000", 2024-09-11T05:52:56.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8853999999999999e-05 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.665164104999999, 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.596413867999999, 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:52:56.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:52:56.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.527504608000001, 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:52:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.88709879, 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:52:56.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:52:56.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157015 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'419, mlcod=99'413)", 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.886090066000001, 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076103459999999996, 2024-09-11T05:52:56.341 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392244+0000", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3500000000000005e-07 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392250+0000", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2650000000000004e-06 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392263+0000", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069e-05 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392269+0000", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7619999999999998e-06 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392311+0000", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2072e-05 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392326+0000", 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4218e-05 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399833+0000", 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075078189999999998 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399854+0000", 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305999999999998e-05 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.784251600000001, 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:52:56.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:52:56.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:52:56.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.783542833999999, 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:52:56.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:52:56.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:52:56.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157101 3.2 e99/83 3:44379fcd:::rbd_data.1124fa04ae41.000000000000003c:head v 99'368, mlcod=99'366)", 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.779088442999999, 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075074139999999996, 2024-09-11T05:52:56.354 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499246+0000", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2020000000000001e-06 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499312+0000", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5473000000000001e-05 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499327+0000", 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4915e-05 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499336+0000", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1330000000000005e-06 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499453+0000", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117045 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499492+0000", 2024-09-11T05:52:56.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8788000000000002e-05 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506706+0000", 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072144399999999999 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506752+0000", 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6418000000000002e-05 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.746960325, 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:52:56.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:52:56.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:52:56.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:52:56.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.364 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:3715 3.c e110/84 3:30a393f2:::rbd_data.11242e6dd3fe.0000000000000054:25 v 110'398, mlcod=109'395)", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.4234789169999997, 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077768000000000004, 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854856+0000", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854867+0000", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1535999999999999e-05 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854893+0000", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5969000000000002e-05 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854914+0000", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645999999999999e-05 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855685+0000", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077148799999999997 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855743+0000", 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7417999999999998e-05 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862514+0000", 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006771628 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862631+0000", 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117105 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.369 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:56.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:52:56.371 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:56.365+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:56.470 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:56.469+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:56.526 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:52:56.549 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:56.549 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:56.549 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:52:56.549 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:56.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:52:56.607 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:56.605+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:56.672 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:56.669+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:56.724 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:56.725 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:52:56.725 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:52:56.725 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:52:56.725 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:56.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:52:56.783 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:56.781+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:56.873 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:56.871+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:56.909 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:52:56.909 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.418760775999999, 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.415468535999999, 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:56.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:52:56.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.415432618000001, 2024-09-11T05:52:56.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:52:56.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410702167, 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.924 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:52:56.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410627214000002, 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:52:56.928 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410560985, 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.410494435, 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.407371199, 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.407280311000001, 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:52:56.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.407205288, 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.950 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.407133882, 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:52:56.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:52:56.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.403003188, 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.959 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:52:56.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.963 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402939912000001, 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:52:56.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402867122, 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.968 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.972 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402794316000001, 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:52:56.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:52:56.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402734183, 2024-09-11T05:52:56.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:52:56.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.402643115, 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.982 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:52:56.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399318463, 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.987 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399207962999999, 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.991 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:52:56.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.399130240000002, 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:52:56.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:52:56.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:52:56.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:52:57.000 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:52:57.001 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:56.997+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:57.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:52:57.103 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:57.101+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:57.134 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.135 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:57.135 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:57.135 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:57.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:52:57.203 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:57.201+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:57.272 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.272 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:57.272 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:57.272 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:57.273 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:57.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:52:57.306 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:57.305+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:57.406 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:57.401+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604298370999999, 2024-09-11T05:52:57.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:57.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:52:57.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604268112, 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.415 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:52:57.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604222827000001, 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.420 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604189961000003, 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:52:57.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:52:57.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604161906000002, 2024-09-11T05:52:57.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:52:57.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.60413484, 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:52:57.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:52:57.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604084839000002, 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.438 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:52:57.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.442 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604052355999997, 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:52:57.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:52:57.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.604023886999997, 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.447 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:52:57.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:52:57.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603995007999998, 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:52:57.451 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:52:57.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:52:57.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.455 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603938190000001, 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:52:57.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:52:57.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603907157999998, 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.460 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:57.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603879139, 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:52:57.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:52:57.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603852209999999, 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.469 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603825643999997, 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.473 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603788432999998, 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:52:57.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:52:57.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603759879999998, 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:52:57.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:52:57.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603711419, 2024-09-11T05:52:57.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:52:57.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:52:57.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 62.603672803999999, 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:52:57.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:52:57.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:165419 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4402181000000001, 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94067595999999998, 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:57.495 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 165419 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966867+0000", 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3560000000000001e-06 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966882+0000", 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966920+0000", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7945999999999999e-05 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966943+0000", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2932e-05 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967101+0000", 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157646 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967123+0000", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2478999999999999e-05 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907405+0000", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94028174200000003 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907540+0000", 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134542 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.499 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.500 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.500 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:57.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:52:57.507 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:57.505+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:57.608 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:57.605+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:57.653 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.653 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:57.653 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:52:57.654 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:57.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:52:57.713 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:57.711+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:57.788 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:52:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:52:57.788 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:52:57.788 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:57.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:52:57.809 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:57.805+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:57.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:52:57.910 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:57.905+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:57.968 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619404583000005, 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:52:57.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619375173999998, 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.974 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:52:57.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619355116999998, 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.978 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619313730000002, 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619281247999993, 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:52:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619261191999996, 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.992 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:52:57.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619241689999996, 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:57.996 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:52:57.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:57.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.000 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619216881, 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.001 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619197732000003, 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:52:58.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.009 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619176917999994, 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:52:58.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619157084999998, 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.018 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619137144999996, 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:58.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.022 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619111024000006, 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:52:58.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:58.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.619074440000006, 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:58.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.614286641000007, 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:52:58.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.614211893999993, 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:52:58.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:52:58.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.614139135000002, 2024-09-11T05:52:58.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:58.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:52:58.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.614118215000005, 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:52:58.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 76.272152023999993, 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:52:58.049 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:52:58.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:52:58.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:52:58.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.563712613, 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.055 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:52:58.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:52:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:52:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:52:58.060 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:52:58.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:52:58.061 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:52:58.062 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:52:58.062 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:52:58.062 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:52:58.062 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:52:58.063 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:52:58.063 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:58.057+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:58.076 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:52:58.077 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:52:58.097 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:52:58.097 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:52:58.118 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:52:58.118 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:52:58.139 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:52:58.139 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:52:58.161 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:52:58.161 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:52:58.161 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:58.159+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:58.179 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:58.179 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:58.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:52:58.284 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:58.283+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:58.354 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:58.354 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:58.363 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:58.359+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:58.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:52:58.469 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:58.467+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:58.545 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:58.545 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:58.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:52:58.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:52:58.605 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:58.603+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:58.665 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:58.661+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:58.730 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:58.730 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:58.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:52:58.789 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:52:58.787+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:52:58.861 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:58.861 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:58.867 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:58.863+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:58.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:52:58.969 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:52:58.967+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:52:59.047 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.047 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:52:59.111 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:59.109+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:59.169 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:59.165+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:52:59.195 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.196 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:52:59.299 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:52:59.297+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:52:59.371 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:59.369+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:59.376 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.376 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:52:59.478 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:52:59.477+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:52:59.549 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:52:59.562 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.562 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.573 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:52:59.571+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:52:59.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:52:59.676 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:52:59.671+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:52:59.751 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.752 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:52:59.810 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:59.809+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:59.875 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:52:59.873+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:52:59.885 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:52:59.885 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:52:59.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:52:59.941 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:52:59.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:52:59.942 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:52:59.942 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-09-11T05:52:59.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0 2024-09-11T05:52:59.987 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:52:59.985+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:00.066 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:00.066 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:00.077 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:00.075+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:00.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:00.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:00.178 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:00.175+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:00.254 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:00.254 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:00.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:00.314 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:00.313+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:00.318 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:00.319 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:00.335 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:00.335 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:00.351 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:00.351 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:00.368 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:00.368 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:00.380 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:00.375+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:00.386 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:00.386 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:00.387 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:00.388 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:00.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:00.406 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:00.407 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:00.426 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:00.426 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:00.444 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:00.445 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:00.487 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:00.485+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:00.515 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.3 primary-affinity to 0 (802) 2024-09-11T05:53:00.577 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:00.577 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:00.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:00.577 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.370516639000002, 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:53:00.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.071337580999995, 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.582 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:109722 3.10 e70/28 3:0d34dbb0:::rbd_data.11242c12cf9f.000000000000003b:head v 70'626, mlcod=70'624)", 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.667712862000002, 2024-09-11T05:53:00.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076668029999999998, 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906633+0000", 2024-09-11T05:53:00.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906634+0000", 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0330000000000001e-06 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906641+0000", 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4200000000000004e-06 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906800+0000", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015904 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.906810+0000", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7710000000000001e-06 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907937+0000", 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001127336 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.907991+0000", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4048000000000002e-05 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914117+0000", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061256369999999997 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:03.914300+0000", 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000183518 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.590 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.188768048, 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:00.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:53:00.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.188165376000001, 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:53:00.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.187218391999998, 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.599 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127919 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'699, mlcod=93'692)", 2024-09-11T05:53:00.603 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.186515513000003, 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079047579999999996, 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387831+0000", 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387833+0000", 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387849+0000", 2024-09-11T05:53:00.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6203000000000001e-05 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387878+0000", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9473000000000001e-05 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387899+0000", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0999999999999999e-05 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391149+0000", 2024-09-11T05:53:00.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032499149999999999 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.391208+0000", 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8615e-05 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395665+0000", 2024-09-11T05:53:00.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044570870000000002 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395736+0000", 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0499000000000004e-05 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127921 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'701, mlcod=93'692)", 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.185063122999999, 2024-09-11T05:53:00.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079345010000000001, 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389283+0000", 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389285+0000", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389301+0000", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389327+0000", 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6142000000000001e-05 2024-09-11T05:53:00.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389344+0000", 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7193999999999998e-05 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394014+0000", 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046697070000000004 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.394073+0000", 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9321999999999999e-05 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397074+0000", 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030010430000000001 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.397218+0000", 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014360299999999999 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.184465287999998, 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:53:00.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:53:00.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127969 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'708, mlcod=93'704)", 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.118652077999997, 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081173600000000005, 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.617 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455694+0000", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455696+0000", 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9139999999999998e-06 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455709+0000", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646000000000001e-05 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455742+0000", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2811999999999997e-05 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.455774+0000", 2024-09-11T05:53:00.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2892999999999998e-05 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456234+0000", 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045955200000000001 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.456269+0000", 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4842000000000003e-05 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463713+0000", 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007443847 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.463812+0000", 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8853999999999999e-05 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:00.621 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.961176909999999, 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:00.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.892426673000003, 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:00.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:53:00.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:53:00.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:53:00.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.823517412999998, 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:53:00.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.183111595, 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:53:00.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157015 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'419, mlcod=99'413)", 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:53:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.182102871000001, 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076103459999999996, 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392243+0000", 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392244+0000", 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3500000000000005e-07 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392250+0000", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2650000000000004e-06 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392263+0000", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3069e-05 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392269+0000", 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7619999999999998e-06 2024-09-11T05:53:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392311+0000", 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2072e-05 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.392326+0000", 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4218e-05 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399833+0000", 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075078189999999998 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399854+0000", 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305999999999998e-05 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:53:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.080264405000001, 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:53:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:53:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.079555638999999, 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:53:00.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157101 3.2 e99/83 3:44379fcd:::rbd_data.1124fa04ae41.000000000000003c:head v 99'368, mlcod=99'366)", 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.075101247999999, 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075074139999999996, 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499245+0000", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499246+0000", 2024-09-11T05:53:00.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2020000000000001e-06 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499312+0000", 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5473000000000001e-05 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499327+0000", 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4915e-05 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499336+0000", 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1330000000000005e-06 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499453+0000", 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117045 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.499492+0000", 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8788000000000002e-05 2024-09-11T05:53:00.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506706+0000", 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072144399999999999 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.506752+0000", 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6418000000000002e-05 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.659 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.04297313, 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.660 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:53:00.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:53:00.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:53:00.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:00.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:3715 3.c e110/84 3:30a393f2:::rbd_data.11242e6dd3fe.0000000000000054:25 v 110'398, mlcod=109'395)", 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.719491722000001, 2024-09-11T05:53:00.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077768000000000004, 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854855+0000", 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854856+0000", 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100000000000001e-06 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854867+0000", 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1535999999999999e-05 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:00.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854893+0000", 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5969000000000002e-05 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.854914+0000", 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645999999999999e-05 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855685+0000", 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077148799999999997 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.855743+0000", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7417999999999998e-05 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862514+0000", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006771628 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:48.862631+0000", 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117105 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.671 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:00.671 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:00.671 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:00.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:00.672 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:00.669+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:00.773 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:00.770+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:00.841 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:00.841 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:00.841 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:00.842 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:00.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:00.903 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:00.901+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:00.974 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:00.973+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:00.979 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:00.979 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:00.980 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:00.980 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:00.980 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:00.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:01.075 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:01.074+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:01.164 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:01.164 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.675027999000001, 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.165 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:53:01.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.671735759000001, 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.170 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:53:01.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.671699840999999, 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:53:01.174 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:53:01.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:53:01.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:53:01.178 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.666969389999998, 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.179 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:53:01.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:53:01.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.666894437, 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:01.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.188 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.666828207999998, 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:53:01.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.666761657999999, 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:01.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.197 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.663638421999998, 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:53:01.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.663547533999999, 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:01.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.663472510999998, 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.207 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.663401104999998, 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.212 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:01.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.216 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.659270411000001, 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.217 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.659207134999999, 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:53:01.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.659134344999998, 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.230 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.659061539, 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:53:01.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:01.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.659001406000002, 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:01.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:53:01.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.658910337999998, 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.240 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:01.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:53:01.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.655585685999998, 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.245 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:53:01.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.655475186, 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.249 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:53:01.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.655397463, 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:53:01.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:53:01.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:01.258 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:01.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:01.260 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:01.257+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:01.361 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:01.358+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:01.391 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:01.392 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:01.392 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:01.392 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:01.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:01.493 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:01.490+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:01.562 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:01.561+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:01.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:53:01.592 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:01.592 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:01.592 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:01.592 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:01.592 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:01.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:01.696 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:01.694+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:01.764 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:01.761+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:01.781 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:01.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.977034478999997, 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:53:01.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.977004219999998, 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:01.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976958934999999, 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.791 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:53:01.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:53:01.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976926069000001, 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:53:01.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976898014, 2024-09-11T05:53:01.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.804 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976870947999998, 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:53:01.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976820946999993, 2024-09-11T05:53:01.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:01.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:53:01.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.814 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976788463999995, 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.819 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976759994999995, 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:53:01.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976731115999996, 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976674298000006, 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.829 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:53:01.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:01.833 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976643265999996, 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:53:01.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:53:01.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:53:01.837 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976615246999998, 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.838 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:53:01.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:01.842 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976588317999997, 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:01.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:01.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:53:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:01.847 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976561751999995, 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:53:01.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:53:01.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976524541000003, 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:01.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976495987999996, 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:53:01.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976447527000005, 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:53:01.861 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:53:01.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.976408911999997, 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:53:01.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:53:01.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:53:01.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:165419 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.8129542080000007, 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94067595999999998, 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:01.871 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 165419 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966867+0000", 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3560000000000001e-06 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966882+0000", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966920+0000", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7945999999999999e-05 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966943+0000", 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2932e-05 2024-09-11T05:53:01.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967101+0000", 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157646 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967123+0000", 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2478999999999999e-05 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:01.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907405+0000", 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94028174200000003 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907540+0000", 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134542 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:01.876 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:01.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:01.878 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:01.874+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:01.979 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:01.974+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:02.039 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:02.040 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:02.040 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:02.040 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:02.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:02.080 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:02.077+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:02.181 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:02.178+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:02.221 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:02.221 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:02.221 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:02.221 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:02.221 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:02.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:02.281 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:02.277+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:02.382 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:02.378+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.403 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052440159, 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.404 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.408 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052410750000007, 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:53:02.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052390693000007, 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:53:02.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052349305999996, 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.418 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:53:02.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:53:02.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052316824000002, 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:53:02.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:53:02.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.427 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052296768000005, 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:53:02.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:53:02.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052277266000004, 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:53:02.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:53:02.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:53:02.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052252456999994, 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:53:02.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052233307999998, 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:53:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:53:02.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052212494000003, 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:53:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052192661000007, 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:02.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:53:02.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052172721000005, 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.455 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:53:02.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:53:02.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:53:02.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.0521466, 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.459 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:53:02.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.052110016, 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.467 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.047322217000001, 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.468 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.472 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.047247470000002, 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:02.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.047174710999997, 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.477 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:02.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.047153791, 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:53:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 80.705187600000002, 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:53:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.996748189000002, 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:53:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:02.497 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:02.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:02.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:02.499 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:02.497+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:02.550 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:53:02.632 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:02.630+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:02.674 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:02.674 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:02.674 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:02.674 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:02.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:02.710 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:02.711 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:02.711 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:02.706+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:02.729 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:02.729 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:02.750 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:02.750 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:02.767 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:02.767 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:02.788 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:02.847 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:02.848 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:02.848 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:02.848 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:02.848 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:02.849 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:02.849 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:02.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:02.851 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:02.851 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:02.852 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:02.849+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:02.953 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:02.953+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:02.968 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:02.968 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:02.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:03.053 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:03.050+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:03.099 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.099 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:03.154 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:03.153+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:03.255 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:03.250+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:03.271 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.271 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:03.373 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:03.370+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:03.454 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.454 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.457 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:03.453+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:03.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:03.557 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:03.554+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:03.608 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.608 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:03.667 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:03.665+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:03.743 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.743 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:03.801 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:03.797+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:03.860 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:03.857+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:03.919 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:03.919 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:03.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:03.961 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:03.958+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:04.062 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:04.058+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:04.098 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.098 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:04.163 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:04.158+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:04.237 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.238 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:04.264 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:04.258+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:04.365 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:04.362+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:04.370 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.370 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:04.466 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:04.462+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:04.509 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.509 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:04.567 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:04.565+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:04.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:53:04.656 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.656 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.667 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:04.665+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:04.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:04.768 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:04.766+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:04.834 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.834 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:04.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:04.870 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:04.866+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:04.971 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:04.971 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:04.971 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:04.966+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:04.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:05.015 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:05.016 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:05.033 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:05.033 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:05.049 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:05.049 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:05.065 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:05.066 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:05.072 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:05.069+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:05.084 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:05.084 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:05.104 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:05.104 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:05.104 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:05.104 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:05.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:05.124 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:05.125 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:05.145 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:05.146 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:05.205 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:05.202+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:05.273 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:05.269+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:05.284 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:05.284 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:05.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:05.284 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:05.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.793096470000002, 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.789804230000001, 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.289 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.789768312, 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:05.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:53:05.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.785037860999999, 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.784962908000001, 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:53:05.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:53:05.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.784896678999999, 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.784830128999999, 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:53:05.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:05.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.781706892999999, 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:53:05.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:53:05.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.781616005, 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:53:05.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:53:05.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:53:05.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.781540981999999, 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.781469575999999, 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:53:05.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.777338881999999, 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:53:05.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.777275606, 2024-09-11T05:53:05.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.777202815999999, 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.77713001, 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:53:05.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:53:05.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.777069876999999, 2024-09-11T05:53:05.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:53:05.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.776978808999999, 2024-09-11T05:53:05.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:53:05.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.773654156999999, 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.359 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:53:05.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:53:05.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:53:05.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.773543657000001, 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:53:05.366 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:05.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.370 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.773465934000001, 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:05.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:53:05.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:53:05.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:05.375 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:05.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:05.377 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:05.373+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:05.477 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:05.474+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:05.507 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:05.507 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:05.507 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:05.507 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:53:05.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:05.546 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:53:05.546 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:53:05.546 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T05:53:05.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T05:53:05.594 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:05.590+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:05.678 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:05.674+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:05.717 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:05.718 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:05.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:05.781 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:05.778+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:05.880 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:05.874+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:05.897 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:05.897 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:05.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:05.897 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091838988999996, 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:53:05.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.902 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091808729999997, 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.906 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091763444999998, 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:53:05.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:53:05.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091730579, 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.911 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.915 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091702523999999, 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:53:05.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:53:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:05.920 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091675457999997, 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:05.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091625457000006, 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:53:05.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:53:05.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091592973999994, 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:53:05.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:53:05.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091564504999994, 2024-09-11T05:53:05.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:53:05.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091535625999995, 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.939 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:53:05.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091478808000005, 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:53:05.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091447775999995, 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:53:05.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091419756999997, 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:05.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:53:05.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091392827999996, 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091366261999994, 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.962 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.966 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091329051000002, 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:53:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091300497999995, 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.971 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.975 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091252037000004, 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:53:05.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:53:05.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.091213421999996, 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.980 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:165419 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.927758718, 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94067595999999998, 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 165419 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966867+0000", 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3560000000000001e-06 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966882+0000", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966920+0000", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7945999999999999e-05 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966943+0000", 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2932e-05 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967101+0000", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157646 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967123+0000", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2478999999999999e-05 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907405+0000", 2024-09-11T05:53:05.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94028174200000003 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907540+0000", 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134542 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:05.989 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:05.990 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:05.990 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 127, 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "created": "2024-09-11T05:50:35.802404+0000", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "modified": "2024-09-11T05:53:05.515417+0000", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "last_up_change": "2024-09-11T05:50:49.084592+0000", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "last_in_change": "2024-09-11T05:52:48.832488+0000", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_num": 5801984, 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_set": [ 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "noscrub", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "pglog_hardlimit", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "purged_snapdirs", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_deletes", 2024-09-11T05:53:05.991 INFO:teuthology.orchestra.run.smithi165.stdout: "sortbitwise" 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_version": 5, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_snapshot": "", 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_max": 3, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "max_osd": 8, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "require_min_compat_client": "luminous", 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "min_compat_client": "luminous", 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "squid", 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "allow_crimson": false, 2024-09-11T05:53:05.992 INFO:teuthology.orchestra.run.smithi165.stdout: "pools": [ 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 1, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": ".mgr", 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:50.566112+0000", 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 1, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool", 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:05.993 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 1, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 1, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 1, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 1, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 1, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:05.994 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "18", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 0, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 0, 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:05.995 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:05.996 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:05.997 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "options": { 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_max": 32, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_min": 1 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": {} 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 6.9000000953674316, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 6.9000000953674316, 2024-09-11T05:53:05.998 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.28999999165534973, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 2, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "rbd", 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:54.170388+0000", 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:53:05.999 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 8, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 8, 2024-09-11T05:53:06.000 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 8, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 8, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 8, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "22", 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:06.001 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 2, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 22, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:06.002 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:06.003 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.619999885559082, 2024-09-11T05:53:06.004 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.619999885559082, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.86000001430511475, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2.25, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2.25, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "pool_client.0", 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:51:04.331649+0000", 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:53:06.005 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 32, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 32, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 32, 2024-09-11T05:53:06.006 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 32, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 32, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "127", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:06.007 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 39, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 127, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:06.008 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:06.009 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 7.4600000381469727, 2024-09-11T05:53:06.010 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 1.75, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 7.4600000381469727, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 1.75, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "error_message": "pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "osds": [ 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:53:06.011 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fdc5357-7501-4582-b301-f5eede8a7bc0", 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6815", 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6817", 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.013 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6818", 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6819", 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.014 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6822", 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6823", 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6820", 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6821", 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6817/4049823921", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6819/4049823921", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6823/4049823921", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6821/4049823921", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "78f30980-40a1-46fd-bc1a-a656b32210a9", 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.017 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6800", 2024-09-11T05:53:06.018 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6801", 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6802", 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6803", 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6806", 2024-09-11T05:53:06.020 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6807", 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.021 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6804", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6805", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6801/2461301418", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6803/2461301418", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6807/2461301418", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6805/2461301418", 2024-09-11T05:53:06.022 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fab9ebc-408e-4bc1-b3ad-385d0e396f6b", 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.023 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6808", 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.024 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6809", 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6810", 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.025 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6811", 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6814", 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.026 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6816", 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6812", 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6813", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6809/3166670912", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6811/3166670912", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6816/3166670912", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6813/3166670912", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.028 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "d3115ede-8d17-4a27-8629-40b19e3c7ee6", 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 118, 2024-09-11T05:53:06.029 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6824", 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6825", 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.030 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6826", 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6827", 2024-09-11T05:53:06.031 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6830", 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6831", 2024-09-11T05:53:06.032 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6828", 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6829", 2024-09-11T05:53:06.033 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6825/3322140846", 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6827/3322140846", 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6831/3322140846", 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6829/3322140846", 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.034 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "7fa88502-3ffd-4aac-924b-0d6c65737ac6", 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 117, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6800", 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6801", 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6802", 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6803", 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.037 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6806", 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6807", 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6804", 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6805", 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6801/347599006", 2024-09-11T05:53:06.039 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6803/347599006", 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6807/347599006", 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6805/347599006", 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "03238cca-4d2b-412d-9fcd-f489cb4a8455", 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 0, 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 0, 2024-09-11T05:53:06.040 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6816", 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6817", 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6818", 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6819", 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6822", 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6823", 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6820", 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6821", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6817/4259095655", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6819/4259095655", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6823/4259095655", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6821/4259095655", 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.045 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "6c46c2d3-7dd1-4b3a-ad76-391c6e624b40", 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0.5262603759765625, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.046 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6824", 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.047 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6825", 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6826", 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.048 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6827", 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6830", 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.049 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6831", 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6828", 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6829", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6825/2366978310", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6827/2366978310", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6831/2366978310", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6829/2366978310", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.051 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "f849014a-7b82-4361-8a23-3b7a0be5f5c0", 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:06.052 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6808", 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6809", 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.053 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6810", 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6811", 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6814", 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6815", 2024-09-11T05:53:06.055 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6812", 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:06.056 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6813", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6809/2576837456", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6811/2576837456", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6815/2576837456", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6813/2576837456", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_xinfo": [ 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.605278+0000", 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.058 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.681171+0000", 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.059 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.531762+0000", 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:46.158901+0000", 2024-09-11T05:53:06.060 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.412535+0000", 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:53:06.061 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 65536, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.537269+0000", 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.062 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.433446+0000", 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.612887+0000", 2024-09-11T05:53:06.063 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap": [], 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_items": [ 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "pgid": "3.1", 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "mappings": [ 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "from": 4, 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: "to": 2 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_primaries": [], 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_temp": [], 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_temp": [], 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklist": {}, 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "range_blocklist": {}, 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profiles": { 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "default": { 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "crush-failure-domain": "osd", 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "k": "2", 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "m": "1", 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin": "jerasure", 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: "technique": "reed_sol_van" 2024-09-11T05:53:06.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps_queue": [ 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "snaps": [ 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "begin": 27, 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "length": 6 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: "new_removed_snaps": [ 2024-09-11T05:53:06.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "snaps": [ 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "begin": 27, 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "length": 1 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "new_purged_snaps": [ 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: "snaps": [ 2024-09-11T05:53:06.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "begin": 33, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "length": 5 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_node_flags": {}, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "device_class_flags": {}, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode": { 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_enabled": false, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_bucket_count": 0, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "degraded_stretch_mode": 0, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "recovering_stretch_mode": 0, 2024-09-11T05:53:06.068 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_bucket": 0 2024-09-11T05:53:06.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.069 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:06.071 INFO:tasks.thrashosds.thrasher:j is {'epoch': 127, 'fsid': 'd5628057-c743-43f5-a51d-1915a16ccfdd', 'created': '2024-09-11T05:50:35.802404+0000', 'modified': '2024-09-11T05:53:05.515417+0000', 'last_up_change': '2024-09-11T05:50:49.084592+0000', 'last_in_change': '2024-09-11T05:52:48.832488+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:50:50.566112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:50:54.170388+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 22, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-11T05:51:04.331649+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 127, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.460000038146973, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 7.460000038146973, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426, 'error_message': 'pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '2fdc5357-7501-4582-b301-f5eede8a7bc0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6815', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 4049823921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 4049823921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 4049823921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 4049823921}]}, 'public_addr': '172.21.15.165:6817/4049823921', 'cluster_addr': '172.21.15.165:6819/4049823921', 'heartbeat_back_addr': '172.21.15.165:6823/4049823921', 'heartbeat_front_addr': '172.21.15.165:6821/4049823921', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '78f30980-40a1-46fd-bc1a-a656b32210a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6800', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2461301418}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2461301418}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 2461301418}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 2461301418}]}, 'public_addr': '172.21.15.165:6801/2461301418', 'cluster_addr': '172.21.15.165:6803/2461301418', 'heartbeat_back_addr': '172.21.15.165:6807/2461301418', 'heartbeat_front_addr': '172.21.15.165:6805/2461301418', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3166670912}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3166670912}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6816', 'nonce': 3166670912}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 3166670912}]}, 'public_addr': '172.21.15.165:6809/3166670912', 'cluster_addr': '172.21.15.165:6811/3166670912', 'heartbeat_back_addr': '172.21.15.165:6816/3166670912', 'heartbeat_front_addr': '172.21.15.165:6813/3166670912', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd3115ede-8d17-4a27-8629-40b19e3c7ee6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 3322140846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 3322140846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 3322140846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 3322140846}]}, 'public_addr': '172.21.15.165:6825/3322140846', 'cluster_addr': '172.21.15.165:6827/3322140846', 'heartbeat_back_addr': '172.21.15.165:6831/3322140846', 'heartbeat_front_addr': '172.21.15.165:6829/3322140846', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 347599006}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 347599006}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 347599006}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 347599006}]}, 'public_addr': '172.21.15.172:6801/347599006', 'cluster_addr': '172.21.15.172:6803/347599006', 'heartbeat_back_addr': '172.21.15.172:6807/347599006', 'heartbeat_front_addr': '172.21.15.172:6805/347599006', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03238cca-4d2b-412d-9fcd-f489cb4a8455', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 4259095655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 4259095655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 4259095655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 4259095655}]}, 'public_addr': '172.21.15.172:6817/4259095655', 'cluster_addr': '172.21.15.172:6819/4259095655', 'heartbeat_back_addr': '172.21.15.172:6823/4259095655', 'heartbeat_front_addr': '172.21.15.172:6821/4259095655', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5262603759765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 2366978310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 2366978310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 2366978310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 2366978310}]}, 'public_addr': '172.21.15.172:6825/2366978310', 'cluster_addr': '172.21.15.172:6827/2366978310', 'heartbeat_back_addr': '172.21.15.172:6831/2366978310', 'heartbeat_front_addr': '172.21.15.172:6829/2366978310', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f849014a-7b82-4361-8a23-3b7a0be5f5c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 2576837456}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2576837456}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2576837456}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 2576837456}]}, 'public_addr': '172.21.15.172:6809/2576837456', 'cluster_addr': '172.21.15.172:6811/2576837456', 'heartbeat_back_addr': '172.21.15.172:6815/2576837456', 'heartbeat_front_addr': '172.21.15.172:6813/2576837456', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.605278+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.681171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.531762+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:46.158901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.412535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.537269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.433446+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.612887+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.1', 'mappings': [{'from': 4, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 27, 'length': 6}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 27, 'length': 1}]}], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 33, 'length': 5}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T05:53:06.071 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.1 2024-09-11T05:53:06.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.1 2024-09-11T05:53:06.072 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:05.989+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:06.091 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:06.089+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:06.192 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:06.189+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:06.245 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:06.246 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:06.246 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:06.292 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:06.289+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:06.373 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:06.373 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:06.374 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:06.374 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:06.374 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:06.437 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:06.434+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:06.494 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:06.490+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164166516999998, 2024-09-11T05:53:06.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:06.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164137108000006, 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:53:06.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:06.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164117051000005, 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.524 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.528 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164075663999995, 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.529 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:53:06.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164043182, 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.534 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:06.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:53:06.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164023126000004, 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.538 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:53:06.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:53:06.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.164003624000003, 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.542 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163978814999993, 2024-09-11T05:53:06.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:53:06.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163959665999997, 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163938852000001, 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.555 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163919019000005, 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.559 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:06.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.563 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163899079000004, 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:53:06.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:53:06.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163872957999999, 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:53:06.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.572 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.163836373999999, 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:53:06.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.159048575, 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:53:06.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.581 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.158973828000001, 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:53:06.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:53:06.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.158901068999995, 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.589 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.158880148999998, 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.590 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.594 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 84.816913958000001, 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:53:06.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:53:06.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:53:06.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.108474547, 2024-09-11T05:53:06.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:06.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:06.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:06.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:06.605 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:06.607 INFO:teuthology.orchestra.run.smithi165.stderr:clear 3.1 pg_upmap_items mapping 2024-09-11T05:53:06.607 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:06.605+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:06.708 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:06.705+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:06.778 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:06.778 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:06.778 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:06.779 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:06.809 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:06.805+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:06.907 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:06.908 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:06.908 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:06.908 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:06.908 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:06.910 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:06.905+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:06.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:07.011 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:07.006+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:07.045 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:07.045 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:07.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.532143132000002, 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.496368949000001, 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.050 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:07.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 72.388895680000005, 2024-09-11T05:53:07.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:53:07.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:53:07.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:53:07.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:53:07.059 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.579472056, 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.060 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:53:07.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.233136560999995, 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:53:07.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:53:07.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.153049128999996, 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.071 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:07.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.152936632999996, 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:157160 3.5 e99/84 3:a03983f5:::rbd_data.1124fa04ae41.0000000000000335:head v 99'478, mlcod=99'476)", 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.465052168, 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081644379999999996, 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578823+0000", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578847+0000", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3952999999999999e-05 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578882+0000", 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5064999999999997e-05 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578899+0000", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7385e-05 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579058+0000", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159023 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579147+0000", 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9183000000000003e-05 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586904+0000", 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007756855 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586985+0000", 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0990999999999998e-05 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [0~393216,425984~32768,491520~32768,557056~32768,622592~32768,688128~32768,753664~32768,819200~32768,884736~32768,950272~32768,1015808~32768,1081344~32768,1146880~32768,1212416~32768,1277952~32768,1343488~32768,1409024~32768,1474560~32768,1540096~32768,1605632~32768,1671168~32768], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T05:53:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.463364742, 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076146820000000002, 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580510+0000", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581661+0000", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011506369999999999 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581714+0000", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3888000000000003e-05 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581761+0000", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6254999999999997e-05 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582042+0000", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028161000000000001 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582089+0000", 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6434999999999998e-05 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.588123+0000", 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060342470000000004 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [1736704~32768,1802240~32768,1867776~32768,1933312~32768,1998848~32768,2064384~32768,2129920~32768,2195456~32768,2260992~32768,2326528~32768,2392064~32768,2457600~32768,2523136~32768,2588672~32768,2654208~32768,2719744~32768,2785280~32768,2850816~32768,2916352~32768,2981888~32768,3047424~32768,3112960~32768,3178496~32768,3244032~32768,3309568~32768,3375104~32768], data_size: 851968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3407872, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.449464377999998, 2024-09-11T05:53:07.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007282078, 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594410+0000", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.716e-06 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595365+0000", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095438399999999996 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595420+0000", 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5568000000000001e-05 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595470+0000", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9431999999999997e-05 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595784+0000", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000313973 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595833+0000", 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9641000000000001e-05 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.601691+0000", 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058573640000000003 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161652 3.1f 3:fb367f8d:::rbd_data.11243956a123.000000000000024c:head [set-alloc-hint object_size 262144 write_size 262144,write 206848~2033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.112064129, 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068935969999999996, 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161652 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931810+0000", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.373e-06 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931818+0000", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931830+0000", 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931836+0000", 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.383e-06 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931895+0000", 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9633000000000003e-05 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932007+0000", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011137900000000001 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932100+0000", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3250000000000003e-05 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.936263+0000", 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041625919999999997 2024-09-11T05:53:07.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938588+0000", 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023251729999999998 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938644+0000", 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6558999999999998e-05 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938702+0000", 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8103000000000003e-05 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164422 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.565634272, 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098766089999999997, 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164422 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478240+0000", 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478247+0000", 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478265+0000", 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7450000000000001e-05 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478275+0000", 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7589999999999992e-06 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478349+0000", 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478497+0000", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147286 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478611+0000", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114585 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.480135+0000", 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001523433 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488026+0000", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078912789999999993 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488077+0000", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1310000000000002e-05 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488115+0000", 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8034000000000002e-05 2024-09-11T05:53:07.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164424 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.565563466, 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099404130000000004, 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164424 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999998e-07 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478314+0000", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1450000000000001e-06 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478324+0000", 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-09-11T05:53:07.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478330+0000", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5799999999999997e-06 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479074+0000", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074402599999999995 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479209+0000", 2024-09-11T05:53:07.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135062 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479298+0000", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8719000000000002e-05 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.481084+0000", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017859569999999999 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488159+0000", 2024-09-11T05:53:07.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070750179999999998 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488209+0000", 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488250+0000", 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0590999999999998e-05 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164434 3.1 3:82a6c363:::rbd_data.11243956a123.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.560488935999999, 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078161930000000008, 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.110 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164434 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483386+0000", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483391+0000", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483403+0000", 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483410+0000", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.464e-06 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483469+0000", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690999999999998e-05 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483589+0000", 2024-09-11T05:53:07.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120588 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483686+0000", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6972999999999995e-05 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.485187+0000", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001501219 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491109+0000", 2024-09-11T05:53:07.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005921595 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491162+0000", 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2735000000000002e-05 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491200+0000", 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8451000000000002e-05 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.115 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164462 3.1 3:83733783:::rbd_data.11243956a123.0000000000000042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.546107040999999, 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081930730000000004, 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.116 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164462 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497767+0000", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497772+0000", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5199999999999999e-06 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497785+0000", 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2874000000000001e-05 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497791+0000", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9449999999999998e-06 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497845+0000", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4784000000000003e-05 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497919+0000", 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3232000000000006e-05 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.498003+0000", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3923000000000003e-05 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.500128+0000", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021251970000000001 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505855+0000", 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057268429999999997 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505906+0000", 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1687999999999997e-05 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505959+0000", 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2760000000000003e-05 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1175 3.4 e108/75 3:2086e591:::rbd_data.1124fa04ae41.00000000000000f4:27 v 108'337, mlcod=107'334)", 2024-09-11T05:53:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.221872466000001, 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010377241000000001, 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822002+0000", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4980000000000001e-06 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822012+0000", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822033+0000", 2024-09-11T05:53:07.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0301000000000001e-05 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822045+0000", 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2775000000000001e-05 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822173+0000", 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012813700000000001 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822229+0000", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5541000000000002e-05 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832284+0000", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010054629000000001 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832378+0000", 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4112999999999998e-05 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.125 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166229 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [watch reconnect cookie 140146595939744 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.5161367749999997, 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0099460929999999, 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.126 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166229 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527738+0000", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.798e-06 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527744+0000", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6420000000000004e-06 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527755+0000", 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443e-05 2024-09-11T05:53:07.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527764+0000", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6149999999999997e-06 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527881+0000", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011735000000000001 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527895+0000", 2024-09-11T05:53:07.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4341e-05 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532195+0000", 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0042995100000001 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532429+0000", 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023393700000000001 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537682+0000", 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052534569999999996 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166230 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.5160374599999997, 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.011669602, 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.131 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166230 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527837+0000", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-06 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527844+0000", 2024-09-11T05:53:07.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527861+0000", 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527871+0000", 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699e-05 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527928+0000", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6870999999999998e-05 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527938+0000", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6229999999999992e-06 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533096+0000", 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005157922 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533210+0000", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011465599999999999 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533714+0000", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050359500000000002 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537700+0000", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039863219999999996 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:07.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539358+0000", 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001657419 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539439+0000", 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1432000000000002e-05 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539505+0000", 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3252 3.1 e127/115 3:82120dc2:::rbd_data.112419b96064.000000000000001c:1b v 127'646, mlcod=125'643)", 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.506489744, 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069382460000000003, 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537384+0000", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537395+0000", 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1185999999999999e-05 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537421+0000", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6489e-05 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537437+0000", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537510+0000", 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3150000000000003e-05 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537544+0000", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4186000000000001e-05 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544250+0000", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067058170000000002 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544322+0000", 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1215000000000002e-05 2024-09-11T05:53:07.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3254 3.1 e127/115 3:85ecff80:::rbd_data.112419b96064.0000000000000003:1b v 127'648, mlcod=125'643)", 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 1.505614671, 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084454270000000001, 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:07.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538259+0000", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0109999999999999e-06 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538268+0000", 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9279999999999999e-06 2024-09-11T05:53:07.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538288+0000", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0194e-05 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538299+0000", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-05 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538366+0000", 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6853000000000005e-05 2024-09-11T05:53:07.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538394+0000", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8075000000000001e-05 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546640+0000", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082464229999999993 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546704+0000", 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3462999999999999e-05 2024-09-11T05:53:07.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:07.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:07.146 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:07.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:07.148 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:07.145+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:07.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:07.249 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:07.245+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:07.311 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:07.311 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:07.312 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:07.312 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:07.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:07.334 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:07.334 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:07.349 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:07.345+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:07.351 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:07.351 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:07.368 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:07.368 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:07.399 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:07.400 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:07.434 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:07.435 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:07.450 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:07.446+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:07.455 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:07.456 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:07.463 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:07.463 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:07.476 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:07.477 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:07.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:07.497 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:07.498 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:07.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:53:07.551 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:07.549+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:07.624 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:07.624 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:07.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:07.687 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:07.685+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:07.752 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:07.749+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:07.769 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:07.769 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:07.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:07.879 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:07.877+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:07.953 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:07.953 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:07.955 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:07.953+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:07.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:08.055 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:08.050+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:08.136 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.136 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:08.194 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:08.193+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:08.257 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:08.254+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:08.268 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.268 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:08.358 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:08.354+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:08.451 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.451 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.459 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:08.454+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:08.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:08.560 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:08.554+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:08.586 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.586 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:08.618 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:53:08.689 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:08.686+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:08.722 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.723 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:08.762 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:08.758+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:08.863 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:08.860+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:08.864 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:08.864 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:08.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:08.965 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:08.962+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:09.051 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:09.051 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:09.066 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:09.060+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:09.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:09.166 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:09.162+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:09.243 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:09.243 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:09.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:09.306 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:09.304+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:09.368 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:09.362+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:09.381 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:09.381 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:09.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:09.469 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:09.466+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:09.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:09.561 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:09.561 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:09.569 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:09.566+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:09.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:09.669 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:09.670 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:09.670 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:09.664+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:09.686 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:09.686 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:09.704 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:09.704 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:09.706 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:09.706 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:09.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:09.721 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:09.721 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:09.742 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:09.742 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:09.762 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:09.762 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:09.771 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:09.768+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:09.783 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:09.783 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:09.804 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:09.804 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:09.838 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:09.838 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:09.838 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:09.838 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:09.838 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:09.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:09.897 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:09.894+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:09.973 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:09.970+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90357 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10207232~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213628467999996, 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096242259999999996, 2024-09-11T05:53:10.018 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90357 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802783+0000", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802784+0000", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9699999999999996e-07 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802787+0000", 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7159999999999999e-06 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802798+0000", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1276e-05 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802805+0000", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0439999999999998e-06 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898666+0000", 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.095860561999999996 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.898821+0000", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015493500000000001 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899026+0000", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020513000000000001 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90358 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10338304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:10.022 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213598208999997, 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096646375000000007, 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90358 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:10.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802814+0000", 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802817+0000", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9660000000000001e-06 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802826+0000", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4100000000000008e-06 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802833+0000", 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1760000000000004e-06 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899128+0000", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096295173999999997 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899267+0000", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013860899999999999 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899460+0000", 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000193353 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90359 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10469376~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213552923999998, 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096898129999999999, 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90359 2024-09-11T05:53:10.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802859+0000", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802860+0000", 2024-09-11T05:53:10.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8000000000000004e-07 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802863+0000", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3129999999999999e-06 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802875+0000", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.135e-05 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802882+0000", 2024-09-11T05:53:10.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9359999999999999e-06 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899544+0000", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096661117000000005 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899635+0000", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1099000000000003e-05 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899757+0000", 2024-09-11T05:53:10.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122435 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90360 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10600448~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213520058, 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097511310000000004, 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.031 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90360 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802892+0000", 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802893+0000", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5300000000000003e-07 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802896+0000", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9399999999999998e-06 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802904+0000", 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802911+0000", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2300000000000002e-06 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.899868+0000", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.096956814000000002 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900115+0000", 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024729400000000001 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900403+0000", 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000287817 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90361 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10731520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213492002999999, 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097752649999999996, 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.035 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90361 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802920+0000", 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802921+0000", 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802923+0000", 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7329999999999999e-06 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802932+0000", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2859999999999999e-06 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802938+0000", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9369999999999998e-06 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900467+0000", 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097528477000000002 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900559+0000", 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1702999999999999e-05 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900673+0000", 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113897 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90362 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10862592~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.039 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213464936999998, 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098336280999999998, 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90362 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802947+0000", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802948+0000", 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7000000000000004e-07 2024-09-11T05:53:10.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802968+0000", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0032000000000001e-05 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802977+0000", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6339999999999996e-06 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802985+0000", 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3520000000000001e-06 2024-09-11T05:53:10.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.900767+0000", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097782047999999996 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901065+0000", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029830900000000002 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901283+0000", 2024-09-11T05:53:10.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021823599999999999 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90363 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 10993664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213414936000007, 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098572146999999999, 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.044 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90363 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802997+0000", 2024-09-11T05:53:10.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.802998+0000", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803001+0000", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9330000000000001e-06 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803012+0000", 2024-09-11T05:53:10.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1895999999999999e-05 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803019+0000", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901363+0000", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098343323999999996 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901457+0000", 2024-09-11T05:53:10.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3919999999999995e-05 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901569+0000", 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011256000000000001 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90364 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11124736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:10.048 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213382452999994, 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098954561999999996, 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90364 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803029+0000", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803030+0000", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9599999999999999e-07 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803033+0000", 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6369999999999999e-06 2024-09-11T05:53:10.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803043+0000", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803050+0000", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7530000000000004e-06 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901664+0000", 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098614361999999997 2024-09-11T05:53:10.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901812+0000", 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147961 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.901984+0000", 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017206400000000001 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.052 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90365 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11255808~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213353983999994, 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099140072999999995, 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90365 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803058+0000", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803059+0000", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803061+0000", 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.864e-06 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803071+0000", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8390000000000002e-06 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803078+0000", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902012+0000", 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098933790999999993 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902094+0000", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2229000000000002e-05 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902198+0000", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010373400000000001 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90366 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11386880~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213325104999996, 2024-09-11T05:53:10.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099500929000000002, 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90366 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803087+0000", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803099+0000", 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1813e-05 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803108+0000", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803117+0000", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8100000000000004e-06 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902284+0000", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099166590999999998 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902426+0000", 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014224499999999999 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902588+0000", 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016190399999999999 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90367 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11517952~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213268287000005, 2024-09-11T05:53:10.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099675557999999997, 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90367 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803144+0000", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5600000000000004e-07 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803148+0000", 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803158+0000", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7329999999999998e-06 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803165+0000", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4599999999999997e-06 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902649+0000", 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099483528000000002 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902725+0000", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5969000000000004e-05 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902819+0000", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4639e-05 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90368 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11649024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:10.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213237254999996, 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100037643, 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90368 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:10.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803175+0000", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803176+0000", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3300000000000001e-07 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803178+0000", 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8930000000000001e-06 2024-09-11T05:53:10.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803187+0000", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803194+0000", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9070000000000001e-06 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.902906+0000", 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099712284999999998 2024-09-11T05:53:10.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903049+0000", 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014310500000000001 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903212+0000", 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000162814 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90369 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11780096~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213209235999997, 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10020775799999999, 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90369 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803203+0000", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1600000000000001e-07 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803206+0000", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5940000000000001e-06 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803214+0000", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4489999999999999e-06 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803221+0000", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8530000000000003e-06 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903239+0000", 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10001731799999999 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903314+0000", 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5319000000000002e-05 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903410+0000", 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6608999999999996e-05 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.072 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90370 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 11911168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213182306999997, 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10056902400000001, 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90370 2024-09-11T05:53:10.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803230+0000", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-07 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803233+0000", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803241+0000", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2160000000000006e-06 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803248+0000", 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8329999999999996e-06 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903494+0000", 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100246473 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903638+0000", 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014365499999999999 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903799+0000", 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160674 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90371 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12042240~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213155740999994, 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10074443800000001, 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.077 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90371 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803256+0000", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803257+0000", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.06e-07 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803259+0000", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5600000000000001e-06 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803268+0000", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.439e-06 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803275+0000", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9040000000000003e-06 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903824+0000", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100549529 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.903904+0000", 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9896999999999998e-05 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904001+0000", 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6502999999999994e-05 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90372 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12173312~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213118530000003, 2024-09-11T05:53:10.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10136518, 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90372 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803293+0000", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803294+0000", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9999999999999997e-07 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803297+0000", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.79e-06 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803306+0000", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0669999999999998e-06 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803313+0000", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9330000000000004e-06 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904086+0000", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100772813 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904231+0000", 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014502499999999999 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904659+0000", 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000427952 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90373 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12304384~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:10.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213089976999996, 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10153701499999999, 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90373 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803322+0000", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803323+0000", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1900000000000002e-07 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803338+0000", 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5224e-05 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803353+0000", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5429000000000001e-05 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803361+0000", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3429999999999998e-06 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904685+0000", 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101324146 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904763+0000", 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8261000000000002e-05 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904859+0000", 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5993000000000001e-05 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90374 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12435456~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213041516000004, 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10190788100000001, 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90374 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803370+0000", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803371+0000", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.37e-07 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803380+0000", 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803391+0000", 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-05 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803400+0000", 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0129999999999999e-06 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.904945+0000", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101545143 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905115+0000", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016935799999999999 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905278+0000", 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000163721 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.093 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90375 3.10 3:0a34dd93:::rbd_data.11249f42baa2.0000000000000000:17 [read 12566528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.213002900999996, 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.102166523, 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90375 2024-09-11T05:53:10.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803409+0000", 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803410+0000", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803412+0000", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803426+0000", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.349e-05 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.803433+0000", 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.126e-06 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905355+0000", 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101921619 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905455+0000", 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100005 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.905576+0000", 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120889 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:165419 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113)", 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.049548197, 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94067595999999998, 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 165419 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966864+0000", 2024-09-11T05:53:10.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966867+0000", 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3560000000000001e-06 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966882+0000", 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966920+0000", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7945999999999999e-05 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.966943+0000", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2932e-05 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967101+0000", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157646 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:10.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:51.967123+0000", 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2478999999999999e-05 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907405+0000", 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94028174200000003 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:52.907540+0000", 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134542 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.103 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.103 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:10.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:10.104 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:10.098+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:10.205 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:10.200+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:10.274 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:10.275 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:10.275 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:10.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:10.337 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:10.334+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:10.406 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:10.402+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:10.412 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:10.413 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:10.413 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:10.413 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:10.413 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:10.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:10.513 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:10.510+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:10.606 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255306254999994, 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.607 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:10.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:53:10.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255276846000001, 2024-09-11T05:53:10.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:53:10.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255256789000001, 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.616 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:53:10.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:53:10.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:53:10.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255215402000005, 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255182919999996, 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.624 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:53:10.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:10.628 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255162863999999, 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255143361999998, 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.633 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:53:10.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:53:10.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255118553000003, 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.637 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.641 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255099404000006, 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:53:10.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:53:10.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:53:10.645 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255078589999997, 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.646 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:10.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:53:10.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.650 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255058757, 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:53:10.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255038816999999, 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.655 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:10.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:53:10.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:53:10.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.255012695999994, 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:53:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:53:10.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.254976111999994, 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:53:10.663 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:53:10.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.250188312999995, 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.668 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.250113565999996, 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.672 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:53:10.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.677 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.250040807000005, 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:53:10.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:53:10.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 90.250019886999993, 2024-09-11T05:53:10.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:53:10.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.688 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 88.908053695999996, 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.689 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:53:10.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:53:10.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 65.199614284999996, 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:53:10.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:53:10.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:10.699 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:10.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:53:10.700 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:10.604+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:10.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:10.750 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:10.746+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:10.810 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:10.804+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:10.873 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:10.873 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:10.874 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:10.874 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:10.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:10.910 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:10.904+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:11.004 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.004 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:11.004 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:11.004 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:11.004 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:11.011 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:11.008+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:11.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:11.112 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:11.106+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:11.137 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.137 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 89.623969617, 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:53:11.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:53:11.142 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 81.588195433999999, 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:53:11.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:53:11.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 76.480722165000003, 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.147 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:53:11.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:53:11.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.671298540999999, 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:53:11.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.324963045999993, 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:53:11.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:11.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.244875613999994, 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.244763117999995, 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:53:11.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:157160 3.5 e99/84 3:a03983f5:::rbd_data.1124fa04ae41.0000000000000335:head v 99'478, mlcod=99'476)", 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.556878652999998, 2024-09-11T05:53:11.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081644379999999996, 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578823+0000", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578847+0000", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3952999999999999e-05 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578882+0000", 2024-09-11T05:53:11.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5064999999999997e-05 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578899+0000", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7385e-05 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579058+0000", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159023 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579147+0000", 2024-09-11T05:53:11.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9183000000000003e-05 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586904+0000", 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007756855 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586985+0000", 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0990999999999998e-05 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [0~393216,425984~32768,491520~32768,557056~32768,622592~32768,688128~32768,753664~32768,819200~32768,884736~32768,950272~32768,1015808~32768,1081344~32768,1146880~32768,1212416~32768,1277952~32768,1343488~32768,1409024~32768,1474560~32768,1540096~32768,1605632~32768,1671168~32768], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.555191227000002, 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076146820000000002, 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580510+0000", 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581661+0000", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011506369999999999 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581714+0000", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3888000000000003e-05 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581761+0000", 2024-09-11T05:53:11.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6254999999999997e-05 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582042+0000", 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028161000000000001 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582089+0000", 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6434999999999998e-05 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.588123+0000", 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060342470000000004 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [1736704~32768,1802240~32768,1867776~32768,1933312~32768,1998848~32768,2064384~32768,2129920~32768,2195456~32768,2260992~32768,2326528~32768,2392064~32768,2457600~32768,2523136~32768,2588672~32768,2654208~32768,2719744~32768,2785280~32768,2850816~32768,2916352~32768,2981888~32768,3047424~32768,3112960~32768,3178496~32768,3244032~32768,3309568~32768,3375104~32768], data_size: 851968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3407872, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.541290863, 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007282078, 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594410+0000", 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.716e-06 2024-09-11T05:53:11.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595365+0000", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095438399999999996 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595420+0000", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5568000000000001e-05 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595470+0000", 2024-09-11T05:53:11.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9431999999999997e-05 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595784+0000", 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000313973 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595833+0000", 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9641000000000001e-05 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.601691+0000", 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058573640000000003 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161652 3.1f 3:fb367f8d:::rbd_data.11243956a123.000000000000024c:head [set-alloc-hint object_size 262144 write_size 262144,write 206848~2033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.203890613999999, 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068935969999999996, 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.186 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161652 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931810+0000", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.373e-06 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931818+0000", 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931830+0000", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931836+0000", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.383e-06 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931895+0000", 2024-09-11T05:53:11.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9633000000000003e-05 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932007+0000", 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011137900000000001 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932100+0000", 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3250000000000003e-05 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.936263+0000", 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041625919999999997 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938588+0000", 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023251729999999998 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938644+0000", 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6558999999999998e-05 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938702+0000", 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8103000000000003e-05 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164422 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.657460756999999, 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098766089999999997, 2024-09-11T05:53:11.192 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164422 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478240+0000", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478247+0000", 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:11.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478265+0000", 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7450000000000001e-05 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478275+0000", 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7589999999999992e-06 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478349+0000", 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478497+0000", 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147286 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478611+0000", 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114585 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.480135+0000", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001523433 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488026+0000", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078912789999999993 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488077+0000", 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1310000000000002e-05 2024-09-11T05:53:11.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488115+0000", 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8034000000000002e-05 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164424 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.657389950999999, 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099404130000000004, 2024-09-11T05:53:11.198 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164424 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999998e-07 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478314+0000", 2024-09-11T05:53:11.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1450000000000001e-06 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478324+0000", 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478330+0000", 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5799999999999997e-06 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479074+0000", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074402599999999995 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479209+0000", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135062 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479298+0000", 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8719000000000002e-05 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.481084+0000", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017859569999999999 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488159+0000", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070750179999999998 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488209+0000", 2024-09-11T05:53:11.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488250+0000", 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0590999999999998e-05 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164434 3.1 3:82a6c363:::rbd_data.11243956a123.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.652315421000001, 2024-09-11T05:53:11.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078161930000000008, 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164434 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:11.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483386+0000", 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483391+0000", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483403+0000", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483410+0000", 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.464e-06 2024-09-11T05:53:11.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483469+0000", 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690999999999998e-05 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483589+0000", 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120588 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483686+0000", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6972999999999995e-05 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.485187+0000", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001501219 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491109+0000", 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005921595 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491162+0000", 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2735000000000002e-05 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491200+0000", 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8451000000000002e-05 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164462 3.1 3:83733783:::rbd_data.11243956a123.0000000000000042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.637933526000001, 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081930730000000004, 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164462 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497767+0000", 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:53:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497772+0000", 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5199999999999999e-06 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497785+0000", 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2874000000000001e-05 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497791+0000", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9449999999999998e-06 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497845+0000", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4784000000000003e-05 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497919+0000", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3232000000000006e-05 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.498003+0000", 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3923000000000003e-05 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.500128+0000", 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021251970000000001 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505855+0000", 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057268429999999997 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505906+0000", 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1687999999999997e-05 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505959+0000", 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2760000000000003e-05 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1175 3.4 e108/75 3:2086e591:::rbd_data.1124fa04ae41.00000000000000f4:27 v 108'337, mlcod=107'334)", 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 24.313698950999999, 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010377241000000001, 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822002+0000", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4980000000000001e-06 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822012+0000", 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822033+0000", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0301000000000001e-05 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822045+0000", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2775000000000001e-05 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822173+0000", 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012813700000000001 2024-09-11T05:53:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822229+0000", 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5541000000000002e-05 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832284+0000", 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010054629000000001 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832378+0000", 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4112999999999998e-05 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166229 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [watch reconnect cookie 140146595939744 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.60796326, 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0099460929999999, 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166229 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527738+0000", 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.798e-06 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527744+0000", 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6420000000000004e-06 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527755+0000", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443e-05 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527764+0000", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6149999999999997e-06 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527881+0000", 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011735000000000001 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527895+0000", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4341e-05 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532195+0000", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0042995100000001 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532429+0000", 2024-09-11T05:53:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023393700000000001 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537682+0000", 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052534569999999996 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166230 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.607863945, 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.011669602, 2024-09-11T05:53:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166230 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527837+0000", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-06 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527844+0000", 2024-09-11T05:53:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527861+0000", 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527871+0000", 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699e-05 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527928+0000", 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6870999999999998e-05 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527938+0000", 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6229999999999992e-06 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533096+0000", 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005157922 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533210+0000", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011465599999999999 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533714+0000", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050359500000000002 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537700+0000", 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039863219999999996 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539358+0000", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001657419 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539439+0000", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1432000000000002e-05 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539505+0000", 2024-09-11T05:53:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3252 3.1 e127/115 3:82120dc2:::rbd_data.112419b96064.000000000000001c:1b v 127'646, mlcod=125'643)", 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5983162289999999, 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069382460000000003, 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537384+0000", 2024-09-11T05:53:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537395+0000", 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1185999999999999e-05 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537421+0000", 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6489e-05 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537437+0000", 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537510+0000", 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3150000000000003e-05 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537544+0000", 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4186000000000001e-05 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544250+0000", 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067058170000000002 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544322+0000", 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1215000000000002e-05 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3254 3.1 e127/115 3:85ecff80:::rbd_data.112419b96064.0000000000000003:1b v 127'648, mlcod=125'643)", 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5974411560000004, 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084454270000000001, 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538259+0000", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0109999999999999e-06 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538268+0000", 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9279999999999999e-06 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538288+0000", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0194e-05 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538299+0000", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-05 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538366+0000", 2024-09-11T05:53:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6853000000000005e-05 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538394+0000", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8075000000000001e-05 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546640+0000", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082464229999999993 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546704+0000", 2024-09-11T05:53:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3462999999999999e-05 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.242 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:11.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:11.244 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:11.240+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:11.344 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:11.340+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:11.411 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.411 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:11.411 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:11.411 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:11.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:11.469 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:11.466+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:11.546 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:11.542+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:11.552 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.552 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:11.552 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:11.552 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:11.552 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:11.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:11.613 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:53:11.614 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:53:11.614 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T05:53:11.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T05:53:11.653 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:53:11.653 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:11.650+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:11.743 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 86.900398010999993, 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:11.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.167781554000001, 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.750 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:11.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:11.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:11.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:11.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.013455682, 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:11.761 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312520587999998, 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.762 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:11.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:11.766 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312509937999998, 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:11.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:11.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312499299000002, 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:11.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312487117000003, 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.775 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:11.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:11.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:11.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312476144000001, 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.782 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:11.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.786 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312465379999999, 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.787 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:11.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312451244000002, 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:11.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:11.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:11.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:11.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:11.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:11.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:11.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:11.801 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.802 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.802 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.802 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.803 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:11.803 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312440713999997, 2024-09-11T05:53:11.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:11.803 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.803 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:11.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:11.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:11.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:11.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:11.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:11.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:11.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.813 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312429033000001, 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.814 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.815 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.815 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:11.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.815 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:11.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:11.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:11.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:11.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:11.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:11.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:11.824 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.312418714000003, 2024-09-11T05:53:11.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:11.825 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.825 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.825 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.825 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:11.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:11.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:11.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:11.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:11.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.835 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.835 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:11.835 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.308905348000003, 2024-09-11T05:53:11.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.837 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:11.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.837 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:11.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:11.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:11.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:11.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:11.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:11.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.308893349000002, 2024-09-11T05:53:11.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.847 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.848 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:11.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.848 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:11.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:11.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:11.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:11.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:11.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.308876974999997, 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:11.857 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.858 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:11.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:11.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:11.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:11.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:11.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:11.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.867 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.308863697, 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.869 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:11.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:11.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.308849803000001, 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:11.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.879 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.289326526, 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:11.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:11.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.885 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.278101895999999, 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.886 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:11.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.891 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:11.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:11.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:11.894 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:11.890+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:11.958 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:11.958 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 133, 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "fsid": "d5628057-c743-43f5-a51d-1915a16ccfdd", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "created": "2024-09-11T05:50:35.802404+0000", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "modified": "2024-09-11T05:53:11.636908+0000", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "last_up_change": "2024-09-11T05:50:49.084592+0000", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "last_in_change": "2024-09-11T05:52:48.832488+0000", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_num": 5804032, 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_set": [ 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "nodeep-scrub", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "pglog_hardlimit", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "purged_snapdirs", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_deletes", 2024-09-11T05:53:11.959 INFO:teuthology.orchestra.run.smithi165.stdout: "sortbitwise" 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_version": 5, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_snapshot": "", 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_max": 3, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "max_osd": 8, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "require_min_compat_client": "luminous", 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "min_compat_client": "jewel", 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "require_osd_release": "squid", 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "allow_crimson": false, 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: "pools": [ 2024-09-11T05:53:11.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 1, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": ".mgr", 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:50.566112+0000", 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 1, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool", 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:11.961 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 1, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 1, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 1, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 1, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 1, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:11.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "18", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 0, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 0, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:11.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:11.964 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:11.965 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "options": { 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_max": 32, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_min": 1 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": {} 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 6.9000000953674316, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 6.9000000953674316, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.28999999165534973, 2024-09-11T05:53:11.966 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 2, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "rbd", 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:50:54.170388+0000", 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:11.967 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 8, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 8, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 8, 2024-09-11T05:53:11.968 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 8, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 8, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "22", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:11.969 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 2, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 22, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:11.970 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:11.971 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:11.972 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 2.619999885559082, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 2.619999885559082, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 0.86000001430511475, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 2.25, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 2.25, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_name": "pool_client.0", 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "create_time": "2024-09-11T05:51:04.331649+0000", 2024-09-11T05:53:11.973 INFO:teuthology.orchestra.run.smithi165.stdout: "flags": 8193, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "type": 1, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 2, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "min_size": 1, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_rule": 0, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_count": 0, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_target": 0, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "is_stretch_pool": false, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "object_hash": 2, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_autoscale_mode": "off", 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num": 32, 2024-09-11T05:53:11.974 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num": 32, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_placement_num_target": 32, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_target": 32, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_num_pending": 32, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_pg_merge_meta": { 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "source_pgid": "0.0", 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "ready_epoch": 0, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_started": 0, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_epoch_clean": 0, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "source_version": "0'0", 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "target_version": "0'0" 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_change": "133", 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend": "0", 2024-09-11T05:53:11.975 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "auid": 0, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_mode": "selfmanaged", 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_seq": 39, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "snap_epoch": 133, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "pool_snaps": [], 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps": "[]", 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_bytes": 0, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "quota_max_objects": 0, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "tiers": [], 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_of": -1, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "read_tier": -1, 2024-09-11T05:53:11.976 INFO:teuthology.orchestra.run.smithi165.stdout: "write_tier": -1, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_mode": "none", 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_bytes": 0, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "target_max_objects": 0, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_flush_age": 0, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "cache_min_evict_age": 0, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profile": "", 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_params": { 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "none" 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_period": 0, 2024-09-11T05:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_count": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "use_gmt_hitset": true, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "min_read_recency_for_promote": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "min_write_recency_for_promote": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "hit_set_search_last_n": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "grade_table": [], 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "stripe_width": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "expected_num_objects": 0, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "fast_read": false, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "options": {}, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "application_metadata": { 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "rbd": {} 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "read_balance": { 2024-09-11T05:53:11.978 INFO:teuthology.orchestra.run.smithi165.stdout: "score_type": "Fair distribution", 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "score_acting": 7.4600000381469727, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "score_stable": 1.75, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "optimal_score": 1, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_acting": 7.4600000381469727, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "raw_score_stable": 1.75, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity_weighted": 0.81578254699707031, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "average_primary_affinity_weighted": 0.81999999284744263, 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "error_message": "pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: "osds": [ 2024-09-11T05:53:11.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fdc5357-7501-4582-b301-f5eede8a7bc0", 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 131, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:11.980 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6815", 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6817", 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.981 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6818", 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6819", 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:11.982 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6822", 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6823", 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:11.983 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6820", 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6821", 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4049823921 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6817/4049823921", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6819/4049823921", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6823/4049823921", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6821/4049823921", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "78f30980-40a1-46fd-bc1a-a656b32210a9", 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:11.985 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.986 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6800", 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6801", 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.987 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6802", 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6803", 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.988 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6806", 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6807", 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.989 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6804", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6805", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2461301418 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6801/2461301418", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6803/2461301418", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6807/2461301418", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6805/2461301418", 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:11.990 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "2fab9ebc-408e-4bc1-b3ad-385d0e396f6b", 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:11.991 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6808", 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6809", 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6810", 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6811", 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.993 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6814", 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6816", 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6812", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6813", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3166670912 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6809/3166670912", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6811/3166670912", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6816/3166670912", 2024-09-11T05:53:11.995 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6813/3166670912", 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "d3115ede-8d17-4a27-8629-40b19e3c7ee6", 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0, 2024-09-11T05:53:11.996 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 118, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6824", 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6825", 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6826", 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:11.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6827", 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6830", 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6831", 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6828", 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.165:6829", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 3322140846 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.165:6825/3322140846", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.165:6827/3322140846", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.165:6831/3322140846", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.165:6829/3322140846", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "7fa88502-3ffd-4aac-924b-0d6c65737ac6", 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 117, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:12.002 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6800", 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6801", 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6802", 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6803", 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6806", 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6807", 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6804", 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6805", 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 347599006 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.006 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6801/347599006", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6803/347599006", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6807/347599006", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6805/347599006", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "03238cca-4d2b-412d-9fcd-f489cb4a8455", 2024-09-11T05:53:12.007 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 23, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.008 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6816", 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6817", 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.009 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6818", 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6819", 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.010 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6822", 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6823", 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.011 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6820", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6821", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 4259095655 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6817/4259095655", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6819/4259095655", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6823/4259095655", 2024-09-11T05:53:12.012 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6821/4259095655", 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "6c46c2d3-7dd1-4b3a-ad76-391c6e624b40", 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 0.5262603759765625, 2024-09-11T05:53:12.013 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6824", 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6825", 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6826", 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6827", 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.016 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6830", 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6831", 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.017 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6828", 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6829", 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2366978310 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6825/2366978310", 2024-09-11T05:53:12.018 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6827/2366978310", 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6831/2366978310", 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6829/2366978310", 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "uuid": "f849014a-7b82-4361-8a23-3b7a0be5f5c0", 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "up": 1, 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "in": 1, 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "weight": 1, 2024-09-11T05:53:12.019 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_affinity": 1, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_begin": 0, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "last_clean_end": 0, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "up_from": 14, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "up_thru": 122, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "down_at": 0, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "lost_at": 0, 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addrs": { 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.020 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6808", 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6809", 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addrs": { 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6810", 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6811", 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addrs": { 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.022 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6814", 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6815", 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addrs": { 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "addrvec": [ 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v2", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6812", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "172.21.15.172:6813", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2576837456 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "public_addr": "172.21.15.172:6809/2576837456", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "cluster_addr": "172.21.15.172:6811/2576837456", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_back_addr": "172.21.15.172:6815/2576837456", 2024-09-11T05:53:12.024 INFO:teuthology.orchestra.run.smithi165.stdout: "heartbeat_front_addr": "172.21.15.172:6813/2576837456", 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "state": [ 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "exists", 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "up" 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_xinfo": [ 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 0, 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.025 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.605278+0000", 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 1, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.681171+0000", 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 2, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.531762+0000", 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 3, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.027 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:46.158901+0000", 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 4, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.412535+0000", 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 5, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 65536, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.537269+0000", 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 6, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.433446+0000", 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": 7, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "down_stamp": "0.000000", 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_probability": 0, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "laggy_interval": 0, 2024-09-11T05:53:12.030 INFO:teuthology.orchestra.run.smithi165.stdout: "features": 4540701547738038271, 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "old_weight": 0, 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "last_purged_snaps_scrub": "2024-09-11T05:50:45.612887+0000", 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "dead_epoch": 0 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap": [], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_items": [], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_upmap_primaries": [], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "pg_temp": [], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "primary_temp": [], 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklist": {}, 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "range_blocklist": {}, 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "erasure_code_profiles": { 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "default": { 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "crush-failure-domain": "osd", 2024-09-11T05:53:12.031 INFO:teuthology.orchestra.run.smithi165.stdout: "k": "2", 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "m": "1", 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "plugin": "jerasure", 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "technique": "reed_sol_van" 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "removed_snaps_queue": [ 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "snaps": [ 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "begin": 24, 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: "length": 5 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: "new_removed_snaps": [ 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: "pool": 3, 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: "snaps": [ 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: "begin": 24, 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: "length": 1 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.033 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "new_purged_snaps": [], 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "crush_node_flags": {}, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "device_class_flags": {}, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode": { 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_enabled": false, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_bucket_count": 0, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "degraded_stretch_mode": 0, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "recovering_stretch_mode": 0, 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: "stretch_mode_bucket": 0 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:12.034 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:12.035 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:12.035 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:12.035 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:12.035 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:12.037 INFO:tasks.thrashosds.thrasher:j is {'epoch': 133, 'fsid': 'd5628057-c743-43f5-a51d-1915a16ccfdd', 'created': '2024-09-11T05:50:35.802404+0000', 'modified': '2024-09-11T05:53:11.636908+0000', 'last_up_change': '2024-09-11T05:50:49.084592+0000', 'last_in_change': '2024-09-11T05:52:48.832488+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:50:50.566112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:50:54.170388+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 22, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-11T05:51:04.331649+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 133, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.460000038146973, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 7.460000038146973, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8157825469970703, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8199999928474426, 'error_message': 'pool 3 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '2fdc5357-7501-4582-b301-f5eede8a7bc0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6815', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6817', 'nonce': 4049823921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6818', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6819', 'nonce': 4049823921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6822', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6823', 'nonce': 4049823921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6820', 'nonce': 4049823921}, {'type': 'v1', 'addr': '172.21.15.165:6821', 'nonce': 4049823921}]}, 'public_addr': '172.21.15.165:6817/4049823921', 'cluster_addr': '172.21.15.165:6819/4049823921', 'heartbeat_back_addr': '172.21.15.165:6823/4049823921', 'heartbeat_front_addr': '172.21.15.165:6821/4049823921', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '78f30980-40a1-46fd-bc1a-a656b32210a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6800', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 2461301418}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6802', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 2461301418}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6806', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 2461301418}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6804', 'nonce': 2461301418}, {'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 2461301418}]}, 'public_addr': '172.21.15.165:6801/2461301418', 'cluster_addr': '172.21.15.165:6803/2461301418', 'heartbeat_back_addr': '172.21.15.165:6807/2461301418', 'heartbeat_front_addr': '172.21.15.165:6805/2461301418', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2fab9ebc-408e-4bc1-b3ad-385d0e396f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6808', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3166670912}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6810', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3166670912}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6814', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6816', 'nonce': 3166670912}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6812', 'nonce': 3166670912}, {'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 3166670912}]}, 'public_addr': '172.21.15.165:6809/3166670912', 'cluster_addr': '172.21.15.165:6811/3166670912', 'heartbeat_back_addr': '172.21.15.165:6816/3166670912', 'heartbeat_front_addr': '172.21.15.165:6813/3166670912', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd3115ede-8d17-4a27-8629-40b19e3c7ee6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6824', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6825', 'nonce': 3322140846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6826', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6827', 'nonce': 3322140846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6830', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6831', 'nonce': 3322140846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6828', 'nonce': 3322140846}, {'type': 'v1', 'addr': '172.21.15.165:6829', 'nonce': 3322140846}]}, 'public_addr': '172.21.15.165:6825/3322140846', 'cluster_addr': '172.21.15.165:6827/3322140846', 'heartbeat_back_addr': '172.21.15.165:6831/3322140846', 'heartbeat_front_addr': '172.21.15.165:6829/3322140846', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7fa88502-3ffd-4aac-924b-0d6c65737ac6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 117, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 347599006}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 347599006}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 347599006}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 347599006}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 347599006}]}, 'public_addr': '172.21.15.172:6801/347599006', 'cluster_addr': '172.21.15.172:6803/347599006', 'heartbeat_back_addr': '172.21.15.172:6807/347599006', 'heartbeat_front_addr': '172.21.15.172:6805/347599006', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '03238cca-4d2b-412d-9fcd-f489cb4a8455', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6817', 'nonce': 4259095655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6818', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6819', 'nonce': 4259095655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 4259095655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6820', 'nonce': 4259095655}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 4259095655}]}, 'public_addr': '172.21.15.172:6817/4259095655', 'cluster_addr': '172.21.15.172:6819/4259095655', 'heartbeat_back_addr': '172.21.15.172:6823/4259095655', 'heartbeat_front_addr': '172.21.15.172:6821/4259095655', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6c46c2d3-7dd1-4b3a-ad76-391c6e624b40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5262603759765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 2366978310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 2366978310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 2366978310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 2366978310}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 2366978310}]}, 'public_addr': '172.21.15.172:6825/2366978310', 'cluster_addr': '172.21.15.172:6827/2366978310', 'heartbeat_back_addr': '172.21.15.172:6831/2366978310', 'heartbeat_front_addr': '172.21.15.172:6829/2366978310', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f849014a-7b82-4361-8a23-3b7a0be5f5c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6809', 'nonce': 2576837456}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6810', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2576837456}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6814', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2576837456}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 2576837456}, {'type': 'v1', 'addr': '172.21.15.172:6813', 'nonce': 2576837456}]}, 'public_addr': '172.21.15.172:6809/2576837456', 'cluster_addr': '172.21.15.172:6811/2576837456', 'heartbeat_back_addr': '172.21.15.172:6815/2576837456', 'heartbeat_front_addr': '172.21.15.172:6813/2576837456', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.605278+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.681171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.531762+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:46.158901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.412535+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.537269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.433446+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T05:50:45.612887+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 24, 'length': 5}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 24, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T05:53:12.037 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-11T05:53:12.037 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:12.032+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:12.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:12.083 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:12.083 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:12.099 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:12.100 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:12.116 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:12.116 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:12.133 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:12.134 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:12.136 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:12.130+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:12.151 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:12.151 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:12.171 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:12.171 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:12.191 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:12.191 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:12.209 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.210 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.212 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:12.212 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:12.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:12.273 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:12.270+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:12.339 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:12.334+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:12.346 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.346 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:12.440 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:12.436+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:12.479 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.479 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:12.541 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:12.538+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:12.641 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:12.638+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:12.658 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.658 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:12.743 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:12.738+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:12.809 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.809 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:12.874 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:12.872+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:12.944 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:12.938+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:12.949 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:12.949 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:12.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:13.045 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:13.042+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:13.128 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.129 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:13.145 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:13.142+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:13.246 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:13.244+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:13.264 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.264 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:13.367 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:13.364+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:13.448 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:13.442+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:13.449 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.449 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:13.549 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:13.546+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:13.633 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.633 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:13.650 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:13.646+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:13.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:53:13.751 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:13.748+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:13.789 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.789 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:13.855 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:13.852+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:13.944 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:13.945 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:13.953 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:13.950+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:13.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:14.054 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:14.050+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:14.129 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:14.129 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:14.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:14.185 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:14.182+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:14.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:14.277 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:14.274+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:14.307 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:14.307 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:14.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:14.356 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:14.352+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:14.368 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:14.369 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:14.385 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:14.386 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:14.404 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:14.404 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:14.420 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:14.420 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:14.441 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:14.441 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:14.456 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:14.454+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:14.462 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:14.462 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:14.479 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:14.479 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:14.479 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:14.482 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:14.483 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:14.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:14.504 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:14.504 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:14.569 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:14.566+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:14.618 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:14.618 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:14.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:14.658 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:14.652+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:14.679 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:53:14.758 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:14.756+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65524 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 32768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460148856000004, 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034387277000000001, 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65524 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348031+0000", 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348039+0000", 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8939999999999993e-06 2024-09-11T05:53:14.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348047+0000", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1470000000000001e-06 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348052+0000", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1320000000000002e-06 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382256+0000", 2024-09-11T05:53:14.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034203725999999997 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382326+0000", 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0216999999999995e-05 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382418+0000", 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1673000000000001e-05 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65525 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 63488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460119446999997, 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034537673999999997, 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65525 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348060+0000", 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7800000000000002e-07 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348063+0000", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3640000000000002e-06 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348069+0000", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.088e-06 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348074+0000", 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8600000000000001e-06 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382448+0000", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034374004 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382516+0000", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8314999999999999e-05 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382598+0000", 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1664999999999998e-05 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65526 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 88064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460099389999996, 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692265, 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65526 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.820 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348080+0000", 2024-09-11T05:53:14.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6100000000000002e-07 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348082+0000", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.995e-06 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348089+0000", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2530000000000001e-06 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348094+0000", 2024-09-11T05:53:14.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1059999999999999e-06 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382626+0000", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034532316 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382692+0000", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6304999999999996e-05 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382772+0000", 2024-09-11T05:53:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929e-05 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65527 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 112640~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460058003, 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034821772000000001, 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65527 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348121+0000", 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348122+0000", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7400000000000003e-07 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348129+0000", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348136+0000", 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348146+0000", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382800+0000", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034654025999999997 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382865+0000", 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4801000000000005e-05 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382943+0000", 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8442000000000006e-05 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65528 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 137216~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460025521000006, 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036885879000000003, 2024-09-11T05:53:14.828 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65528 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348154+0000", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348157+0000", 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.204e-06 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348163+0000", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1299999999999998e-06 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348168+0000", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8860000000000003e-06 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.382970+0000", 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034802380000000001 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.383037+0000", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6649000000000005e-05 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385040+0000", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020031749999999998 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65529 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:14.832 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.460005464999995, 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037068630999999998, 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65529 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348174+0000", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348175+0000", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3900000000000005e-07 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348176+0000", 2024-09-11T05:53:14.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.902e-06 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348182+0000", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348187+0000", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1619999999999999e-06 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385099+0000", 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036911509000000002 2024-09-11T05:53:14.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385164+0000", 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5617999999999994e-05 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385243+0000", 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8219000000000004e-05 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.836 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65530 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 217088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459985962999994, 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037238886999999998, 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65530 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348193+0000", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348194+0000", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.96e-07 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348199+0000", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348205+0000", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2349999999999996e-06 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348212+0000", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4180000000000002e-06 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385273+0000", 2024-09-11T05:53:14.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037061430999999999 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385344+0000", 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1033999999999998e-05 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385432+0000", 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8129999999999998e-05 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65531 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 241664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459961153999998, 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037413343000000002, 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65531 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348218+0000", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348219+0000", 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7e-07 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348221+0000", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348227+0000", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8889999999999999e-06 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348231+0000", 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7570000000000004e-06 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385474+0000", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037242614 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385545+0000", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1086000000000003e-05 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385632+0000", 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6654000000000001e-05 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.844 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65532 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 266240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459942005000002, 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037599270999999997, 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65532 2024-09-11T05:53:14.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348237+0000", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348238+0000", 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8099999999999998e-07 2024-09-11T05:53:14.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348240+0000", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9659999999999999e-06 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348246+0000", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348251+0000", 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.87e-06 2024-09-11T05:53:14.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385674+0000", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037423670999999999 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385750+0000", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5450000000000004e-05 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385837+0000", 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6932e-05 2024-09-11T05:53:14.848 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65533 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459921191000007, 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037781124999999999, 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65533 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348258+0000", 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348259+0000", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000005e-07 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348261+0000", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9369999999999998e-06 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348267+0000", 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348272+0000", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385881+0000", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037609947999999997 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.385954+0000", 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2247000000000001e-05 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386039+0000", 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5641000000000002e-05 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65534 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 346112~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.853 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459901357999996, 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037903183, 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65534 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:14.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348278+0000", 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348279+0000", 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3300000000000002e-07 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348281+0000", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9319999999999999e-06 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348287+0000", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348292+0000", 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386078+0000", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037785879000000001 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386131+0000", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3399999999999997e-05 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386181+0000", 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0077000000000002e-05 2024-09-11T05:53:14.857 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65535 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459881417999995, 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038045795, 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.858 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65535 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348298+0000", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3999999999999997e-07 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348305+0000", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4099999999999996e-06 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348312+0000", 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5859999999999999e-06 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348317+0000", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1209999999999998e-06 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386202+0000", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037884371999999999 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386266+0000", 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3894999999999997e-05 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386344+0000", 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8071000000000005e-05 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65536 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 395264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459855297000004, 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038150168999999998, 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65536 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348324+0000", 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348325+0000", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7300000000000002e-07 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348327+0000", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0279999999999999e-06 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348332+0000", 2024-09-11T05:53:14.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9229999999999999e-06 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348353+0000", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0034000000000002e-05 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386371+0000", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038018926000000001 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386425+0000", 2024-09-11T05:53:14.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3152999999999997e-05 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386474+0000", 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9731999999999998e-05 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65537 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 450560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.459818713000004, 2024-09-11T05:53:14.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038267018999999999, 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65537 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348361+0000", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0800000000000004e-07 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348363+0000", 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1150000000000001e-06 2024-09-11T05:53:14.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348370+0000", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1510000000000001e-06 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.348374+0000", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8659999999999997e-06 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386491+0000", 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038116941000000001 2024-09-11T05:53:14.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386555+0000", 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3798000000000001e-05 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.386628+0000", 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2539999999999993e-05 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.870 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65559 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 555008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.455030914000005, 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034465605000000003, 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65559 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353149+0000", 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1600000000000001e-07 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353151+0000", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9819999999999998e-06 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353157+0000", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.207e-06 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353162+0000", 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.853e-06 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387471+0000", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034308933999999999 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387542+0000", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0523000000000002e-05 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387614+0000", 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2589999999999994e-05 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65561 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.454956167000006, 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034491186, 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65561 2024-09-11T05:53:14.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353223+0000", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353224+0000", 2024-09-11T05:53:14.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6499999999999999e-07 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353227+0000", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353235+0000", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8609999999999998e-06 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353255+0000", 2024-09-11T05:53:14.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0373999999999998e-05 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387632+0000", 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034377029000000003 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387668+0000", 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5920999999999997e-05 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387714+0000", 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6411e-05 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65563 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 604160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.454883408000001, 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034510383999999998, 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.879 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65563 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353296+0000", 2024-09-11T05:53:14.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353297+0000", 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9800000000000004e-07 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353299+0000", 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1579999999999999e-06 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353305+0000", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1979999999999998e-06 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353310+0000", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4620000000000004e-06 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387730+0000", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034419532000000003 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387764+0000", 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3733999999999999e-05 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387806+0000", 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2802000000000001e-05 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:65564 3.19 3:98a44b54:::rbd_data.1124b2f65e38.0000000000000014:11 [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:14.883 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.454862488000003, 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034692540000000001, 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 65564 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353317+0000", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353318+0000", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4499999999999997e-07 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353319+0000", 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8190000000000001e-06 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353325+0000", 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9610000000000002e-06 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.353330+0000", 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0080000000000002e-06 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387857+0000", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034526326000000003 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.387928+0000", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1670999999999999e-05 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:40.388009+0000", 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1210000000000003e-05 2024-09-11T05:53:14.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:77971 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 93.112896297000006, 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054247785, 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.888 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 77971 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695283+0000", 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695286+0000", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2499999999999998e-06 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695298+0000", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695329+0000", 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0355999999999999e-05 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695351+0000", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2608000000000001e-05 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695525+0000", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017350199999999999 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.695672+0000", 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014686300000000001 2024-09-11T05:53:14.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.703698+0000", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080267910000000001 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749340+0000", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045641908000000002 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749417+0000", 2024-09-11T05:53:14.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6163999999999998e-05 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749475+0000", 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8471000000000003e-05 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.749531+0000", 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.596e-05 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:119086 3.19 3.b32e119 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e72)", 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.404456886000006, 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10506362700000001, 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 119086 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:14.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403723+0000", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.403725+0000", 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0940000000000002e-06 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404370+0000", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064570000000000003 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404388+0000", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7224e-05 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404397+0000", 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1200000000000008e-06 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404505+0000", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108084 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.404522+0000", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6722e-05 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508774+0000", 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10425288000000001 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:14.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:05.508786+0000", 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1803e-05 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:14.898 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:14.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:14.900 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:14.896+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:15.000 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:14.996+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:15.020 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:15.020 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:15.020 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:15.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:15.100 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:15.096+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:15.149 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.149 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:15.149 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:15.149 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:15.149 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:15.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:15.209 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:15.206+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:53:15.285 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 93.771454018, 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:15.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:53:15.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:53:15.289 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.735679834999999, 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.290 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:53:15.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 80.628206566000003, 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:53:15.294 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.818782941999999, 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:53:15.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:53:15.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 74.472447446999993, 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.306 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:15.311 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 74.392360014999994, 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:15.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:53:15.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:53:15.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.315 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 74.392247518999994, 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:157160 3.5 e99/84 3:a03983f5:::rbd_data.1124fa04ae41.0000000000000335:head v 99'478, mlcod=99'476)", 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.704363053999998, 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081644379999999996, 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578823+0000", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578847+0000", 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3952999999999999e-05 2024-09-11T05:53:15.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578882+0000", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5064999999999997e-05 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578899+0000", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7385e-05 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579058+0000", 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159023 2024-09-11T05:53:15.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579147+0000", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9183000000000003e-05 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586904+0000", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007756855 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586985+0000", 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0990999999999998e-05 2024-09-11T05:53:15.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [0~393216,425984~32768,491520~32768,557056~32768,622592~32768,688128~32768,753664~32768,819200~32768,884736~32768,950272~32768,1015808~32768,1081344~32768,1146880~32768,1212416~32768,1277952~32768,1343488~32768,1409024~32768,1474560~32768,1540096~32768,1605632~32768,1671168~32768], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.702675628000001, 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076146820000000002, 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580510+0000", 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581661+0000", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011506369999999999 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581714+0000", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3888000000000003e-05 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581761+0000", 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6254999999999997e-05 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582042+0000", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028161000000000001 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582089+0000", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6434999999999998e-05 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.588123+0000", 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060342470000000004 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [1736704~32768,1802240~32768,1867776~32768,1933312~32768,1998848~32768,2064384~32768,2129920~32768,2195456~32768,2260992~32768,2326528~32768,2392064~32768,2457600~32768,2523136~32768,2588672~32768,2654208~32768,2719744~32768,2785280~32768,2850816~32768,2916352~32768,2981888~32768,3047424~32768,3112960~32768,3178496~32768,3244032~32768,3309568~32768,3375104~32768], data_size: 851968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3407872, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.688775264, 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007282078, 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594410+0000", 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.716e-06 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595365+0000", 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095438399999999996 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595420+0000", 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5568000000000001e-05 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595470+0000", 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9431999999999997e-05 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595784+0000", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000313973 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595833+0000", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9641000000000001e-05 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.601691+0000", 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058573640000000003 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161652 3.1f 3:fb367f8d:::rbd_data.11243956a123.000000000000024c:head [set-alloc-hint object_size 262144 write_size 262144,write 206848~2033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 31.351375014999999, 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068935969999999996, 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161652 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931810+0000", 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.373e-06 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931818+0000", 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931830+0000", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931836+0000", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.383e-06 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931895+0000", 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9633000000000003e-05 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932007+0000", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011137900000000001 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932100+0000", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3250000000000003e-05 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.936263+0000", 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041625919999999997 2024-09-11T05:53:15.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938588+0000", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023251729999999998 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938644+0000", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6558999999999998e-05 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938702+0000", 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8103000000000003e-05 2024-09-11T05:53:15.337 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164422 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.804945157999999, 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098766089999999997, 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.338 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164422 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478240+0000", 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478247+0000", 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478265+0000", 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7450000000000001e-05 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478275+0000", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7589999999999992e-06 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478349+0000", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478497+0000", 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147286 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478611+0000", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114585 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.480135+0000", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001523433 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488026+0000", 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078912789999999993 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488077+0000", 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1310000000000002e-05 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488115+0000", 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8034000000000002e-05 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164424 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.804874351999999, 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099404130000000004, 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164424 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:15.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999998e-07 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478314+0000", 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1450000000000001e-06 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478324+0000", 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478330+0000", 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5799999999999997e-06 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479074+0000", 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074402599999999995 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479209+0000", 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135062 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479298+0000", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8719000000000002e-05 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.481084+0000", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017859569999999999 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488159+0000", 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070750179999999998 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488209+0000", 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488250+0000", 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0590999999999998e-05 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164434 3.1 3:82a6c363:::rbd_data.11243956a123.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.799799822000001, 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078161930000000008, 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164434 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483386+0000", 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483391+0000", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483403+0000", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483410+0000", 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.464e-06 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483469+0000", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690999999999998e-05 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483589+0000", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120588 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483686+0000", 2024-09-11T05:53:15.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6972999999999995e-05 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.485187+0000", 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001501219 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491109+0000", 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005921595 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491162+0000", 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2735000000000002e-05 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491200+0000", 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8451000000000002e-05 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164462 3.1 3:83733783:::rbd_data.11243956a123.0000000000000042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:15.355 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.785417927000001, 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081930730000000004, 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164462 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497767+0000", 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:53:15.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497772+0000", 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5199999999999999e-06 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497785+0000", 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2874000000000001e-05 2024-09-11T05:53:15.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497791+0000", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9449999999999998e-06 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497845+0000", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4784000000000003e-05 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497919+0000", 2024-09-11T05:53:15.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3232000000000006e-05 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.498003+0000", 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3923000000000003e-05 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.500128+0000", 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021251970000000001 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505855+0000", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057268429999999997 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505906+0000", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1687999999999997e-05 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505959+0000", 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2760000000000003e-05 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1175 3.4 e108/75 3:2086e591:::rbd_data.1124fa04ae41.00000000000000f4:27 v 108'337, mlcod=107'334)", 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.461183351999999, 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010377241000000001, 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822002+0000", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4980000000000001e-06 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822012+0000", 2024-09-11T05:53:15.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822033+0000", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0301000000000001e-05 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822045+0000", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2775000000000001e-05 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822173+0000", 2024-09-11T05:53:15.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012813700000000001 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822229+0000", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5541000000000002e-05 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832284+0000", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010054629000000001 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832378+0000", 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4112999999999998e-05 2024-09-11T05:53:15.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166229 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [watch reconnect cookie 140146595939744 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.755447661, 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0099460929999999, 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.366 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166229 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527738+0000", 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.798e-06 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527744+0000", 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6420000000000004e-06 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527755+0000", 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443e-05 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527764+0000", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6149999999999997e-06 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527881+0000", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011735000000000001 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527895+0000", 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4341e-05 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532195+0000", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0042995100000001 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532429+0000", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023393700000000001 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537682+0000", 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052534569999999996 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.370 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166230 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 14.755348346, 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.011669602, 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166230 2024-09-11T05:53:15.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527837+0000", 2024-09-11T05:53:15.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-06 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527844+0000", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527861+0000", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527871+0000", 2024-09-11T05:53:15.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699e-05 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527928+0000", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6870999999999998e-05 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527938+0000", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6229999999999992e-06 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533096+0000", 2024-09-11T05:53:15.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005157922 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533210+0000", 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011465599999999999 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533714+0000", 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050359500000000002 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537700+0000", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039863219999999996 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539358+0000", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001657419 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539439+0000", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1432000000000002e-05 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539505+0000", 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3252 3.1 e127/115 3:82120dc2:::rbd_data.112419b96064.000000000000001c:1b v 127'646, mlcod=125'643)", 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.7458006299999997, 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069382460000000003, 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.377 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537384+0000", 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537395+0000", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1185999999999999e-05 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537421+0000", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6489e-05 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537437+0000", 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537510+0000", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3150000000000003e-05 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537544+0000", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4186000000000001e-05 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544250+0000", 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067058170000000002 2024-09-11T05:53:15.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544322+0000", 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1215000000000002e-05 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3254 3.1 e127/115 3:85ecff80:::rbd_data.112419b96064.0000000000000003:1b v 127'648, mlcod=125'643)", 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 9.7449255570000002, 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084454270000000001, 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.381 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538259+0000", 2024-09-11T05:53:15.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0109999999999999e-06 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538268+0000", 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9279999999999999e-06 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538288+0000", 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0194e-05 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538299+0000", 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-05 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538366+0000", 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6853000000000005e-05 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538394+0000", 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8075000000000001e-05 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546640+0000", 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082464229999999993 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546704+0000", 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3462999999999999e-05 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.385 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:15.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:15.387 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:15.382+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:15.488 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:15.484+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3384 3.18 e137/122 3:18496c36:::rbd_data.112420bb439a.00000000000000b1:14 v 137'411, mlcod=135'408)", 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:15.569295+0000", 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0025624380000000002, 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026115370000000001, 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.572 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569295+0000", 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569295+0000", 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569297+0000", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.863e-06 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569311+0000", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4311000000000001e-05 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569342+0000", 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0737000000000001e-05 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569360+0000", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8063e-05 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569433+0000", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3133000000000006e-05 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.569455+0000", 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2031e-05 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3385 3.18 e137/122 3:19d22cff:::rbd_data.112420bb439a.000000000000000c:14 v 137'413, mlcod=135'408)", 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:15.570394+0000", 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0014633000000000001, 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001616237, 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570394+0000", 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570394+0000", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570396+0000", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.629e-06 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570409+0000", 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3331e-05 2024-09-11T05:53:15.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570431+0000", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2154000000000002e-05 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570442+0000", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0694e-05 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570501+0000", 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9184999999999997e-05 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:15.570520+0000", 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9222e-05 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-09-11T05:53:15.579 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:15.589 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:15.586+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:15.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:15.689 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:15.684+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:15.765 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.765 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:15.766 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:15.766 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:15.766 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:15.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:15.790 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:15.788+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:15.891 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:15.886+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:15.909 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:15.909 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:15.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:15.909 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.064754214000004, 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:15.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:15.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.332137756999998, 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:15.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:15.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:15.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.921 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.177811884999997, 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:15.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.926 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476876791000002, 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.927 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476866141000002, 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:15.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:15.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476855501999999, 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.47684332, 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.940 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:15.944 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476832346999998, 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:15.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:15.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:15.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476821583000003, 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:15.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476807446999999, 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:15.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476796917000001, 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:15.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:15.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476785235999998, 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:15.962 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.966 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.476774917, 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:15.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:15.970 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.473261551, 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.971 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.473249551999999, 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.975 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:15.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.473233178000001, 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:15.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.473219899999997, 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.984 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:15.988 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.473206005999998, 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:15.992 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.453682729000001, 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.993 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:15.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:15.998 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.442458099, 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:15.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:16.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:16.004 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:16.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:16.006 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:16.000+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:16.107 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:16.104+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:16.137 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:16.137 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:16.137 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:16.137 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:16.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:16.238 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:16.236+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:16.308 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:16.304+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:16.317 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:16.317 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:16.317 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:16.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:16.409 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:16.406+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.891425980000001, 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:16.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.891318075999997, 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:53:16.505 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.891259757, 2024-09-11T05:53:16.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:16.513 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.891205634999999, 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.517 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890215376999997, 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:16.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:53:16.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:53:16.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.521 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890167228000003, 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:53:16.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890134961999998, 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.526 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890104948999998, 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.530 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:16.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890076016999998, 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:53:16.534 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.890034610999997, 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:53:16.538 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:53:16.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:53:16.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.542 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886388191999998, 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:16.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:53:16.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.546 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886305384000003, 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:16.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.550 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886228381999999, 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:53:16.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:53:16.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886156894000003, 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886100931999998, 2024-09-11T05:53:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:53:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:53:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.886061685999998, 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:53:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.885988337999997, 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.885958592000001, 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.882071375000002, 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:53:16.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.882016987, 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:53:16.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:16.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:16.584 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:16.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:16.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:16.586 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:16.580+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:16.686 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:16.682+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:16.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:53:16.747 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:16.747 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:16.765 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:16.765 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:16.774 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:16.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:16.774 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:16.774 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:16.782 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:16.782 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:16.787 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:16.782+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:16.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:16.801 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:16.801 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:16.822 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:16.823 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:16.841 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:16.841 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:16.860 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:16.860 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:16.879 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:16.879 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:16.889 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:16.882+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:16.917 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:16.917 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:16.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:16.990 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:16.986+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:17.038 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:53:17.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:53:17.038 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-11T05:53:17.038 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T05:53:17.039 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T05:53:17.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-11T05:53:17.051 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.051 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:17.090 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:17.086+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:17.163 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:17.163 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-09-11T05:53:17.163 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:17.187 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.187 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:17.221 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:17.218+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:17.292 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:17.286+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:17.326 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.326 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:17.393 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:17.388+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:17.459 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.459 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:17.494 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:17.490+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:17.594 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:17.592+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:17.639 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.639 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:17.695 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:17.690+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:17.698 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:53:17.795 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:17.790+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:17.829 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.829 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:17.896 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:17.892+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:17.967 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:17.968 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:17.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:18.029 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:18.026+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:18.098 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:18.096+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:18.109 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.109 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:18.209 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:18.206+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:18.287 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.288 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.299 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:18.294+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:18.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:18.401 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:18.400+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:18.461 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.461 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:18.501 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:18.496+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:18.596 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.596 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.602 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:18.598+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:18.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:18.702 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:18.698+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:18.783 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.783 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:18.845 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:18.842+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:18.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:18.945 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:18.942+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:18.980 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:18.980 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:18.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:19.004 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:19.000+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:19.040 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:19.041 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:19.056 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:19.057 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:19.074 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:19.075 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:19.091 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:19.091 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:19.104 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:19.102+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:19.112 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:19.112 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:19.132 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:19.133 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:19.150 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:19.150 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:19.150 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:19.150 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:19.153 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:19.153 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:19.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:19.174 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:19.174 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:19.241 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:19.238+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:19.289 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:19.289 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:19.289 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:19.289 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:19.289 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:19.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:19.305 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:19.300+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:19.407 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:19.404+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:53:19.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 97.950474193000005, 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:53:19.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 89.914700010000004, 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:19.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 84.807226740999994, 2024-09-11T05:53:19.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:53:19.480 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 83.997803117000004, 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.481 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:53:19.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:19.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 78.651467621999998, 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:53:19.486 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:53:19.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:53:19.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 78.571380189999999, 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:53:19.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 78.571267693999999, 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:53:19.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:157160 3.5 e99/84 3:a03983f5:::rbd_data.1124fa04ae41.0000000000000335:head v 99'478, mlcod=99'476)", 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.883383229000003, 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081644379999999996, 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.500 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578823+0000", 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-11T05:53:19.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578847+0000", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3952999999999999e-05 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578882+0000", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5064999999999997e-05 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578899+0000", 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7385e-05 2024-09-11T05:53:19.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579058+0000", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159023 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579147+0000", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9183000000000003e-05 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586904+0000", 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007756855 2024-09-11T05:53:19.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586985+0000", 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0990999999999998e-05 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [0~393216,425984~32768,491520~32768,557056~32768,622592~32768,688128~32768,753664~32768,819200~32768,884736~32768,950272~32768,1015808~32768,1081344~32768,1146880~32768,1212416~32768,1277952~32768,1343488~32768,1409024~32768,1474560~32768,1540096~32768,1605632~32768,1671168~32768], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.881695803, 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076146820000000002, 2024-09-11T05:53:19.504 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580510+0000", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581661+0000", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011506369999999999 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581714+0000", 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3888000000000003e-05 2024-09-11T05:53:19.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581761+0000", 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6254999999999997e-05 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582042+0000", 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028161000000000001 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582089+0000", 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6434999999999998e-05 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.588123+0000", 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060342470000000004 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.508 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [1736704~32768,1802240~32768,1867776~32768,1933312~32768,1998848~32768,2064384~32768,2129920~32768,2195456~32768,2260992~32768,2326528~32768,2392064~32768,2457600~32768,2523136~32768,2588672~32768,2654208~32768,2719744~32768,2785280~32768,2850816~32768,2916352~32768,2981888~32768,3047424~32768,3112960~32768,3178496~32768,3244032~32768,3309568~32768,3375104~32768], data_size: 851968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3407872, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.867795438999998, 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007282078, 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594410+0000", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.716e-06 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595365+0000", 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095438399999999996 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595420+0000", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5568000000000001e-05 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595470+0000", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9431999999999997e-05 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595784+0000", 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000313973 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595833+0000", 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9641000000000001e-05 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.601691+0000", 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058573640000000003 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161652 3.1f 3:fb367f8d:::rbd_data.11243956a123.000000000000024c:head [set-alloc-hint object_size 262144 write_size 262144,write 206848~2033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:53:19.512 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.53039519, 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068935969999999996, 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161652 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:19.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931810+0000", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.373e-06 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931818+0000", 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-11T05:53:19.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931830+0000", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931836+0000", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.383e-06 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931895+0000", 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9633000000000003e-05 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932007+0000", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011137900000000001 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932100+0000", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3250000000000003e-05 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.936263+0000", 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041625919999999997 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938588+0000", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023251729999999998 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938644+0000", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6558999999999998e-05 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938702+0000", 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8103000000000003e-05 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.517 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164422 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.983965333, 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098766089999999997, 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164422 2024-09-11T05:53:19.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478240+0000", 2024-09-11T05:53:19.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478247+0000", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478265+0000", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7450000000000001e-05 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478275+0000", 2024-09-11T05:53:19.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7589999999999992e-06 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478349+0000", 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478497+0000", 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147286 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:19.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478611+0000", 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114585 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.480135+0000", 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001523433 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488026+0000", 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078912789999999993 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488077+0000", 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1310000000000002e-05 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488115+0000", 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8034000000000002e-05 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164424 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:19.523 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.983894526999997, 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099404130000000004, 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164424 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999998e-07 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478314+0000", 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1450000000000001e-06 2024-09-11T05:53:19.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478324+0000", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478330+0000", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5799999999999997e-06 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479074+0000", 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074402599999999995 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479209+0000", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135062 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479298+0000", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8719000000000002e-05 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.481084+0000", 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017859569999999999 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488159+0000", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070750179999999998 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488209+0000", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488250+0000", 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0590999999999998e-05 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164434 3.1 3:82a6c363:::rbd_data.11243956a123.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.978819997000002, 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078161930000000008, 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.529 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164434 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483386+0000", 2024-09-11T05:53:19.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483391+0000", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483403+0000", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483410+0000", 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.464e-06 2024-09-11T05:53:19.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483469+0000", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690999999999998e-05 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483589+0000", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120588 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483686+0000", 2024-09-11T05:53:19.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6972999999999995e-05 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.485187+0000", 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001501219 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491109+0000", 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005921595 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491162+0000", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2735000000000002e-05 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491200+0000", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8451000000000002e-05 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164462 3.1 3:83733783:::rbd_data.11243956a123.0000000000000042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:19.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.964438102000003, 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081930730000000004, 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164462 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497767+0000", 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497772+0000", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5199999999999999e-06 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497785+0000", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2874000000000001e-05 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497791+0000", 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9449999999999998e-06 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497845+0000", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4784000000000003e-05 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497919+0000", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3232000000000006e-05 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.498003+0000", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3923000000000003e-05 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.500128+0000", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021251970000000001 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505855+0000", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057268429999999997 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505906+0000", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1687999999999997e-05 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505959+0000", 2024-09-11T05:53:19.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2760000000000003e-05 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1175 3.4 e108/75 3:2086e591:::rbd_data.1124fa04ae41.00000000000000f4:27 v 108'337, mlcod=107'334)", 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.640203526999997, 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010377241000000001, 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822002+0000", 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4980000000000001e-06 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822012+0000", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822033+0000", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0301000000000001e-05 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822045+0000", 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2775000000000001e-05 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822173+0000", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012813700000000001 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822229+0000", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5541000000000002e-05 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832284+0000", 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010054629000000001 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832378+0000", 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4112999999999998e-05 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166229 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [watch reconnect cookie 140146595939744 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.934467836, 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0099460929999999, 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.544 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166229 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527738+0000", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.798e-06 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527744+0000", 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6420000000000004e-06 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527755+0000", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443e-05 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527764+0000", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6149999999999997e-06 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527881+0000", 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011735000000000001 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527895+0000", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4341e-05 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532195+0000", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0042995100000001 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532429+0000", 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023393700000000001 2024-09-11T05:53:19.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537682+0000", 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052534569999999996 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166230 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.934368521, 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.011669602, 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.549 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166230 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:19.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527837+0000", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-06 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527844+0000", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527861+0000", 2024-09-11T05:53:19.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527871+0000", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699e-05 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527928+0000", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6870999999999998e-05 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527938+0000", 2024-09-11T05:53:19.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6229999999999992e-06 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533096+0000", 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005157922 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533210+0000", 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011465599999999999 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533714+0000", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050359500000000002 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537700+0000", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039863219999999996 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539358+0000", 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001657419 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539439+0000", 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1432000000000002e-05 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539505+0000", 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3252 3.1 e127/115 3:82120dc2:::rbd_data.112419b96064.000000000000001c:1b v 127'646, mlcod=125'643)", 2024-09-11T05:53:19.555 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.924820805, 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069382460000000003, 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:19.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537384+0000", 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537395+0000", 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1185999999999999e-05 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537421+0000", 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6489e-05 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537437+0000", 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537510+0000", 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3150000000000003e-05 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537544+0000", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4186000000000001e-05 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544250+0000", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067058170000000002 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544322+0000", 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1215000000000002e-05 2024-09-11T05:53:19.559 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3254 3.1 e127/115 3:85ecff80:::rbd_data.112419b96064.0000000000000003:1b v 127'648, mlcod=125'643)", 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.923945732, 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084454270000000001, 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:19.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538259+0000", 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0109999999999999e-06 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538268+0000", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9279999999999999e-06 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538288+0000", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0194e-05 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538299+0000", 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-05 2024-09-11T05:53:19.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538366+0000", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6853000000000005e-05 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538394+0000", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8075000000000001e-05 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546640+0000", 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082464229999999993 2024-09-11T05:53:19.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546704+0000", 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3462999999999999e-05 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:19.564 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:19.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:19.566 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:19.560+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:19.667 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:19.662+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:19.692 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:19.692 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:19.692 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:19.692 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:19.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:19.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:53:19.801 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:19.798+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:19.868 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:19.864+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:19.887 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:19.887 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:19.887 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:19.887 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:19.887 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:19.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:19.997 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:19.994+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.226259810000002, 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:20.071 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:20.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:20.076 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.493643353000003, 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:20.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:20.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:20.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.082 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.339317481000002, 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.083 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.087 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638382387, 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.088 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:20.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:20.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:20.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638371737, 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638361097999997, 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.096 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638348915999998, 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.101 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:20.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638337943000003, 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:20.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638327179000001, 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638313042999997, 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.118 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638302512999999, 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:20.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638290832000003, 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.638280512999998, 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.127 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:20.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:20.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:20.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.634767146999998, 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.131 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:20.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.634755147999996, 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.139 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.634738773999999, 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.140 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.634725496000001, 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.144 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:20.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.148 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.634711602000003, 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:20.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.615188324999998, 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:20.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.603963694999997, 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.159 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:20.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:20.164 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:20.165 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:20.068+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:20.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:20.211 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:20.208+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:20.273 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:20.270+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:20.296 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:20.296 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:20.296 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:20.296 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:20.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:20.373 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:20.370+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:20.474 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:20.470+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:20.480 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:20.480 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:20.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:20.524 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:53:20.575 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:20.572+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:20.634 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:20.634 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.026803155000003, 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:53:20.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:53:20.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:53:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.026695251, 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:53:20.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:20.643 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.026636932000002, 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:20.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.026582810000001, 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:53:20.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025592551999999, 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:53:20.652 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.656 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025544402999998, 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025512137, 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:53:20.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:53:20.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025482124, 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:53:20.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:53:20.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:53:20.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025453192000001, 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.670 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:53:20.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.025411785999999, 2024-09-11T05:53:20.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021765367, 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:53:20.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021682558999998, 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:53:20.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021605557000001, 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.687 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:53:20.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:53:20.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021534068999998, 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:53:20.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021478107, 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.699 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021438861, 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.700 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021365512999999, 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.704 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:53:20.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.021335766999997, 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.709 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.017448549999997, 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.713 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:53:20.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:53:20.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:53:20.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.017394162000002, 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:20.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:20.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:20.721 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:20.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:20.723 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:20.718+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:20.824 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:20.820+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:20.899 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:20.900 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:20.900 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:20.900 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:20.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:20.924 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:20.918+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:21.025 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:21.020+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:21.078 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:21.078 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:21.078 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:21.078 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:21.078 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:21.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:21.135 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:21.132+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:21.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:21.227 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:21.224+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.737043275999994, 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:53:21.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.737011101999997, 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:53:21.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736988081999996, 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.265 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:21.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736951598000005, 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:53:21.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736924326999997, 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.274 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:21.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.278 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.7369013, 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736877585000002, 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:53:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736854769999994, 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:53:21.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736831811000002, 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:53:21.291 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:53:21.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:53:21.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736808886000006, 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:53:21.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:53:21.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736785171999998, 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:53:21.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:53:21.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:21.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736758476000006, 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:21.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:53:21.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:53:21.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736732110000005, 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:53:21.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736697414999995, 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:53:21.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:53:21.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.736518126999997, 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.318 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:53:21.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:53:21.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.733867446000005, 2024-09-11T05:53:21.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:53:21.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.733845396000007, 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:53:21.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:53:21.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:53:21.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.733632270000001, 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.331 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:53:21.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:53:21.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:21.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.733602106999996, 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:53:21.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:53:21.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.733578210000005, 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:21.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:21.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:53:21.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:21.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:21.344 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:21.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:21.346 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:21.342+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:21.349 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:21.349 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:21.365 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:21.365 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:21.381 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:21.381 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:21.397 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:21.397 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:21.416 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:21.416 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:21.435 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:21.435 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:21.447 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:21.444+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:21.453 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:21.453 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:21.471 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:21.471 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:21.514 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:21.514 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:21.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:21.548 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:21.542+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:21.648 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:21.644+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:21.696 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:21.696 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:21.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:21.750 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:21.746+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:21.850 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:21.848+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:21.885 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:21.885 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:21.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:21.951 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:21.946+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:22.052 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:22.048+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:22.077 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.077 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:22.178 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:53:22.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:53:22.178 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T05:53:22.178 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-11T05:53:22.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-11T05:53:22.179 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:22.176+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:22.254 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:22.250+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:22.256 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.256 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:22.358 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:22.356+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:22.433 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.433 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:22.493 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:22.490+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:22.557 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:22.554+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:22.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:53:22.579 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.579 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:22.659 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:22.656+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:22.733 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T05:53:22.759 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:22.756+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:22.778 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.778 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:22.885 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:22.882+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:22.959 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:22.959 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:22.960 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:22.956+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:22.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:23.062 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:23.056+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:23.130 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:23.131 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:23.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:23.163 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:23.160+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:23.263 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:23.263 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:23.264 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:23.260+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:23.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:23.365 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:23.360+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:23.391 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:23.391 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:23.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:23.465 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:23.460+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:23.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:23.525 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:23.525 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:23.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:23.585 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:23.582+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:23.622 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:23.622 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:23.638 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:23.639 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:23.655 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:23.655 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:23.667 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:23.664+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:23.673 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:23.673 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:23.693 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:23.693 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:23.697 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:23.698 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:23.698 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:23.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:23.714 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:23.714 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:23.734 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:23.735 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:23.740 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:53:23.763 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:23.764 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:23.769 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:23.764+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:23.846 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:77873 3.4 e52/23 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:head v 52'143, mlcod=52'140)", 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 102.33298180200001, 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012964954000000001, 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511730+0000", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.511733+0000", 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6769999999999999e-06 2024-09-11T05:53:23.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514599+0000", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028660199999999999 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514627+0000", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7994999999999999e-05 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514641+0000", 2024-09-11T05:53:23.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4321999999999999e-05 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514718+0000", 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7309000000000001e-05 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.514744+0000", 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6143e-05 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524617+0000", 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098725819999999995 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:41.524695+0000", 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7905999999999996e-05 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:84929 3.4 3:26b39cf9:::rbd_data.1124c865af2a.000000000000000a:13 [read 245760~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T05:53:23.851 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 94.297207619000005, 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070786369999999996, 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 84929 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547504+0000", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547505+0000", 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4999999999999998e-07 2024-09-11T05:53:23.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547507+0000", 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7369999999999998e-06 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547515+0000", 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4100000000000002e-06 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.547520+0000", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0089999999999998e-06 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554218+0000", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066980629999999998 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554304+0000", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6618000000000003e-05 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:49.554583+0000", 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027835000000000002 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:89798 3.0 3:0453c8ff:::rbd_data.11242d2cfbf4.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 89.189734349999995, 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010793568, 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 89798 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654977+0000", 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654980+0000", 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7769999999999998e-06 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.654993+0000", 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3169e-05 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655025+0000", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1928999999999999e-05 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655040+0000", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5073e-05 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655120+0000", 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9729999999999997e-05 2024-09-11T05:53:23.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.655243+0000", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122631 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.656867+0000", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001624856 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665565+0000", 2024-09-11T05:53:23.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086977679999999998 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665600+0000", 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4363999999999998e-05 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665699+0000", 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9556999999999996e-05 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:54.665771+0000", 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1713999999999998e-05 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:90700 3.0 3:0343b65b:::rbd_data.11242d2cfbf4.0000000000000092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 88.380310726000005, 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082580559999999997, 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 90700 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464401+0000", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464402+0000", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4339999999999999e-06 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464410+0000", 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0830000000000004e-06 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464452+0000", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1634999999999997e-05 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.464469+0000", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.698e-05 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466026+0000", 2024-09-11T05:53:23.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015564210000000001 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466235+0000", 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020991599999999999 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.466516+0000", 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028028699999999999 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.470614+0000", 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040979190000000002 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472461+0000", 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018476580000000001 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472593+0000", 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013152199999999999 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:55.472659+0000", 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6200999999999999e-05 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94442 3.4 3:24891913:::rbd_data.112419b96064.00000000000000cf:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 83.033975230999999, 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024107488999999999, 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.868 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94442 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810736+0000", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810738+0000", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8929999999999999e-06 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810746+0000", 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.746e-06 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810764+0000", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7873999999999999e-05 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810776+0000", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2028000000000001e-05 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810841+0000", 2024-09-11T05:53:23.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499000000000004e-05 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.810963+0000", 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121443 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.814881+0000", 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039185610000000001 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834647+0000", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019765718000000002 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834710+0000", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2807999999999999e-05 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834786+0000", 2024-09-11T05:53:23.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5500000000000006e-05 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.834844+0000", 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8418999999999998e-05 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94505 3.0 3:00542219:::rbd_data.112419b96064.0000000000000063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:23.874 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.953887799, 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071816930000000003, 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94505 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890824+0000", 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890826+0000", 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.21e-06 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890838+0000", 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712999999999999e-05 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890868+0000", 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0205e-05 2024-09-11T05:53:23.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890888+0000", 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446999999999999e-05 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890969+0000", 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0815000000000005e-05 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891107+0000", 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013819099999999999 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898006+0000", 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068981119999999996 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:94509 3.0 3:051e595e:::rbd_data.112419b96064.0000000000000067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T05:53:23.879 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.953775303, 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073923069999999999, 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 94509 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890936+0000", 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890938+0000", 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3740000000000001e-06 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890945+0000", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1559999999999998e-06 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890966+0000", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1512999999999999e-05 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.890986+0000", 2024-09-11T05:53:23.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0006000000000001e-05 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891615+0000", 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062845000000000002 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.891985+0000", 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037021599999999998 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:00.898329+0000", 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063435920000000003 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.4388.0:157160 3.5 e99/84 3:a03983f5:::rbd_data.1124fa04ae41.0000000000000335:head v 99'478, mlcod=99'476)", 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 45.265890837999997, 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081644379999999996, 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.884 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578821+0000", 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578823+0000", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9829999999999999e-06 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578847+0000", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3952999999999999e-05 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578882+0000", 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5064999999999997e-05 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.578899+0000", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7385e-05 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579058+0000", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159023 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.579147+0000", 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9183000000000003e-05 2024-09-11T05:53:23.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586904+0000", 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007756855 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:38.586985+0000", 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0990999999999998e-05 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [0~393216,425984~32768,491520~32768,557056~32768,622592~32768,688128~32768,753664~32768,819200~32768,884736~32768,950272~32768,1015808~32768,1081344~32768,1146880~32768,1212416~32768,1277952~32768,1343488~32768,1409024~32768,1474560~32768,1540096~32768,1605632~32768,1671168~32768], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.264203412000001, 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076146820000000002, 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580508+0000", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.580510+0000", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.61e-06 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581661+0000", 2024-09-11T05:53:23.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011506369999999999 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581714+0000", 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3888000000000003e-05 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.581761+0000", 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6254999999999997e-05 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582042+0000", 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028161000000000001 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.582089+0000", 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6434999999999998e-05 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.588123+0000", 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060342470000000004 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(3.1 104/103 [PushOp(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8, version: 36'65, data_included: [1736704~32768,1802240~32768,1867776~32768,1933312~32768,1998848~32768,2064384~32768,2129920~32768,2195456~32768,2260992~32768,2326528~32768,2392064~32768,2457600~32768,2523136~32768,2588672~32768,2654208~32768,2719744~32768,2785280~32768,2850816~32768,2916352~32768,2981888~32768,3047424~32768,3112960~32768,3178496~32768,3244032~32768,3309568~32768,3375104~32768], data_size: 851968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:83fd7e63:::rbd_data.11246b7d9e18.0000000000000003:8@36'65, size: 3407872, copy_subset: [(0, 3407872)], clone_subset: {}, snapset: 8=[]:{8: [8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3407872, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1703936, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.250303047999999, 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007282078, 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594409+0000", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.594410+0000", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.716e-06 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595365+0000", 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095438399999999996 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595420+0000", 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5568000000000001e-05 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595470+0000", 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9431999999999997e-05 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595784+0000", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000313973 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.595833+0000", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9641000000000001e-05 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.601691+0000", 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058573640000000003 2024-09-11T05:53:23.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161652 3.1f 3:fb367f8d:::rbd_data.11243956a123.000000000000024c:head [set-alloc-hint object_size 262144 write_size 262144,write 206848~2033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.912902799000001, 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068935969999999996, 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161652 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931809+0000", 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931810+0000", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.373e-06 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931818+0000", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5809999999999999e-06 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931830+0000", 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931836+0000", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.383e-06 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.931895+0000", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9633000000000003e-05 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932007+0000", 2024-09-11T05:53:23.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011137900000000001 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.932100+0000", 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3250000000000003e-05 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.936263+0000", 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041625919999999997 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938588+0000", 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023251729999999998 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938644+0000", 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6558999999999998e-05 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.938702+0000", 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8103000000000003e-05 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.902 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164422 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.366472942000001, 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098766089999999997, 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164422 2024-09-11T05:53:23.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478239+0000", 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478240+0000", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.28e-06 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478247+0000", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478265+0000", 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7450000000000001e-05 2024-09-11T05:53:23.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478275+0000", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7589999999999992e-06 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478349+0000", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478497+0000", 2024-09-11T05:53:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147286 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478611+0000", 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114585 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.480135+0000", 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001523433 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488026+0000", 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078912789999999993 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488077+0000", 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1310000000000002e-05 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488115+0000", 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8034000000000002e-05 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.908 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164424 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.366402135999998, 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099404130000000004, 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164424 2024-09-11T05:53:23.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478310+0000", 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999998e-07 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478314+0000", 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1450000000000001e-06 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478324+0000", 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.478330+0000", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5799999999999997e-06 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479074+0000", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074402599999999995 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479209+0000", 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135062 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.479298+0000", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8719000000000002e-05 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.481084+0000", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017859569999999999 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488159+0000", 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070750179999999998 2024-09-11T05:53:23.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488209+0000", 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.488250+0000", 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0590999999999998e-05 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164434 3.1 3:82a6c363:::rbd_data.11243956a123.0000000000000025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.361327606000003, 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078161930000000008, 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164434 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483384+0000", 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483386+0000", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309999999999999e-06 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483391+0000", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5230000000000003e-06 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483403+0000", 2024-09-11T05:53:23.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1523e-05 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483410+0000", 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.464e-06 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483469+0000", 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8690999999999998e-05 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483589+0000", 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120588 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.483686+0000", 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6972999999999995e-05 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.485187+0000", 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001501219 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491109+0000", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005921595 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491162+0000", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2735000000000002e-05 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.491200+0000", 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8451000000000002e-05 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.920 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:164462 3.1 3:83733783:::rbd_data.11243956a123.0000000000000042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.346945710999996, 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081930730000000004, 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.921 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164462 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497766+0000", 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497767+0000", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497772+0000", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5199999999999999e-06 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497785+0000", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2874000000000001e-05 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497791+0000", 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9449999999999998e-06 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497845+0000", 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4784000000000003e-05 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.497919+0000", 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3232000000000006e-05 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.498003+0000", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3923000000000003e-05 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.500128+0000", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021251970000000001 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505855+0000", 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057268429999999997 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505906+0000", 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1687999999999997e-05 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.505959+0000", 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2760000000000003e-05 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.926 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1175 3.4 e108/75 3:2086e591:::rbd_data.1124fa04ae41.00000000000000f4:27 v 108'337, mlcod=107'334)", 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 37.022711135999998, 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010377241000000001, 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822001+0000", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822002+0000", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4980000000000001e-06 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822012+0000", 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822033+0000", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0301000000000001e-05 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822045+0000", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2775000000000001e-05 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822173+0000", 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012813700000000001 2024-09-11T05:53:23.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.822229+0000", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5541000000000002e-05 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832284+0000", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010054629000000001 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:46.832378+0000", 2024-09-11T05:53:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4112999999999998e-05 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166229 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [watch reconnect cookie 140146595939744 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.316975445000001, 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0099460929999999, 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166229 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527736+0000", 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527738+0000", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.798e-06 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527744+0000", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6420000000000004e-06 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527755+0000", 2024-09-11T05:53:23.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443e-05 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527764+0000", 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6149999999999997e-06 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527881+0000", 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011735000000000001 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:23.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527895+0000", 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4341e-05 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532195+0000", 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0042995100000001 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.532429+0000", 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023393700000000001 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537682+0000", 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052534569999999996 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:166230 3.5 3:a092e70b:::rbd_header.11243e6dddb9:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.316876130000001, 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.011669602, 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166230 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527836+0000", 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527837+0000", 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.277e-06 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527844+0000", 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527861+0000", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527871+0000", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0699e-05 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527928+0000", 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6870999999999998e-05 2024-09-11T05:53:23.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:00.527938+0000", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6229999999999992e-06 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533096+0000", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005157922 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533210+0000", 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011465599999999999 2024-09-11T05:53:23.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.533714+0000", 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050359500000000002 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.537700+0000", 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039863219999999996 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:23.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539358+0000", 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001657419 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539439+0000", 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1432000000000002e-05 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:01.539505+0000", 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.942 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3252 3.1 e127/115 3:82120dc2:::rbd_data.112419b96064.000000000000001c:1b v 127'646, mlcod=125'643)", 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.307328414000001, 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069382460000000003, 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:23.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537383+0000", 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537384+0000", 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9000000000000002e-07 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537395+0000", 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1185999999999999e-05 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537421+0000", 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6489e-05 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537437+0000", 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537510+0000", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3150000000000003e-05 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.537544+0000", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4186000000000001e-05 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544250+0000", 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067058170000000002 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.544322+0000", 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1215000000000002e-05 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.0.0:3254 3.1 e127/115 3:85ecff80:::rbd_data.112419b96064.0000000000000003:1b v 127'648, mlcod=125'643)", 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 18.306453341000001, 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084454270000000001, 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:23.947 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538258+0000", 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538259+0000", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0109999999999999e-06 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538268+0000", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9279999999999999e-06 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538288+0000", 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0194e-05 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538299+0000", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.048e-05 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538366+0000", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6853000000000005e-05 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.538394+0000", 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8075000000000001e-05 2024-09-11T05:53:23.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546640+0000", 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082464229999999993 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:53:05.546704+0000", 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3462999999999999e-05 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:23.951 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:23.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:23.953 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:23.948+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:24.054 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:24.052+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:24.073 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:24.073 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:24.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:24.154 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:24.152+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:24.206 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:24.207 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:24.207 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:24.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:24.265 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:24.262+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 99.495007629, 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.339 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:24.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 61.762391172000001, 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:24.344 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:24.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:24.350 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 60.6080653, 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.351 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907130205999998, 2024-09-11T05:53:24.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:24.360 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907119555999998, 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.361 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.365 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907108917000002, 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:24.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:24.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:24.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907096735000003, 2024-09-11T05:53:24.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:24.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:24.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907085762000001, 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:24.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907074997999999, 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:24.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:24.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:24.387 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907060862000002, 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:24.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907050331999997, 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:24.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:24.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907038651000001, 2024-09-11T05:53:24.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:24.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.907028332000003, 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:24.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:24.405 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.903514966000003, 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:24.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:24.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:24.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.903502967000001, 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.410 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.903486592999997, 2024-09-11T05:53:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.903473314999999, 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.903459421000001, 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.883936144000003, 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.872711514000002, 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:24.440 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:24.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:24.442 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:24.436+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:24.543 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:24.540+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:24.572 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:24.572 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:24.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:24.679 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:24.676+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:24.744 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:24.740+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:24.760 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:24.761 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:24.761 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:24.761 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:24.761 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:24.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:24.867 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:24.864+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:24.946 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:24.942+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.340556597000003, 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.340448692999999, 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:53:24.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:53:24.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:53:24.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:53:24.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.340390374000002, 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.957 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:53:24.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.340336252, 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:53:24.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339345993999999, 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.966 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:53:24.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:53:24.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339297844999997, 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:53:24.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339265578999999, 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:53:24.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:24.980 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339235565999999, 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:53:24.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:53:24.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:53:24.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339206634, 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:53:24.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.339165227999999, 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:53:24.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.994 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335518809, 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:24.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:53:24.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335436000999998, 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:24.999 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:53:25.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335358999, 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.003 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:53:25.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:53:25.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335287510999997, 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:53:25.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:53:25.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:53:25.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335231549, 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:53:25.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335192302999999, 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:53:25.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335118954999999, 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:53:25.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.335089209000003, 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.030 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.331201991999997, 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.331147604000002, 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:25.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.039 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:25.083 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:25.080+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:25.148 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:25.144+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:25.212 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:25.212 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:25.212 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:25.212 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:25.271 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:25.268+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:25.350 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:25.346+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:25.450 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:25.450+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:25.537 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:25.537 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:25.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:25.537 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019297906999995, 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.538 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:53:25.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:53:25.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019265732999997, 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:53:25.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019242712999997, 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:53:25.547 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019206229000005, 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:53:25.552 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:53:25.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:53:25.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:53:25.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019178957999998, 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:25.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:53:25.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019155931, 2024-09-11T05:53:25.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:25.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:53:25.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.566 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019132216000003, 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:53:25.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019109400999994, 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:53:25.571 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:25.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:25.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019086442000003, 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019063517000006, 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:53:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:53:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019039802999998, 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:53:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.019013107000006, 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:53:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:53:25.596 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.597 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.018986741000006, 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.601 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:53:25.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.605 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.018952045999995, 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:53:25.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:25.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.018772757999997, 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.610 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:53:25.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.016122077000006, 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.616 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:53:25.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:53:25.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:53:25.620 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.016100026999993, 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.621 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:53:25.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:53:25.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.015886901000002, 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:25.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.015856737999997, 2024-09-11T05:53:25.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:53:25.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.015832841000005, 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:25.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:25.639 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:25.641 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:25.636+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:25.741 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:25.736+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:25.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:53:25.783 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:25.784 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:25.784 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:25.784 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:25.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:25.869 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:25.866+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:25.942 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:25.938+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:25.967 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:25.967 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:25.967 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:25.968 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:25.968 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:25.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:25.984 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:25.984 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:26.002 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:26.002 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:26.021 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:26.022 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:26.038 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:26.038 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:26.043 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:26.038+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:26.056 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:26.057 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:26.076 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:26.076 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:26.096 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:26.096 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:26.105 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.105 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:26.161 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:26.158+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:26.242 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.242 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.245 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:26.240+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:26.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:26.346 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:26.340+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:26.420 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.420 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:26.447 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:26.442+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:26.548 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:26.546+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:26.601 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.601 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:26.649 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:26.644+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:26.748 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.748 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.749 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:26.746+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:26.758 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:53:26.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:26.869 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:26.866+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:26.951 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:26.948+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:26.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:27.052 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:27.048+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:27.130 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.131 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:27.189 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:27.186+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:27.254 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:27.250+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:27.266 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.266 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:27.373 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:27.370+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:27.451 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.452 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.457 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:27.454+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:27.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:27.557 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:27.554+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:27.631 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.631 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:27.689 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:27.686+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:27.759 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:27.756+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:27.771 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7, 0] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-11T05:53:27.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T05:53:27.772 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.9759786050177582 2024-09-11T05:53:27.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 0.9759786050177582 2024-09-11T05:53:27.832 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.832 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:27.859 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:27.856+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:27.960 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:27.954+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:27.976 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:27.976 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:27.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:28.061 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:28.056+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:28.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:28.147 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:28.147 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:28.161 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:28.158+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:28.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:28.254 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:28.254 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:28.263 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:28.260+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:28.272 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:28.272 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:28.289 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:28.289 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:28.306 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:28.306 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:28.322 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:28.322 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:28.323 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-11T05:53:28.323 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:28.325 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:28.326 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:28.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:28.346 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:28.347 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:28.363 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:28.360+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:28.367 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:28.367 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:28.388 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:28.388 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:28.448 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:28.448 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-11T05:53:28.448 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-11T05:53:28.448 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-11T05:53:28.448 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:28.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:28.463 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:28.460+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:28.564 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:28.558+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 103.779094973, 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.046478515999993, 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 64.892152644000006, 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191217549999998, 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:28.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191206899999997, 2024-09-11T05:53:28.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:28.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:28.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.649 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191196261000002, 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.650 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:28.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191184079000003, 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.654 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:28.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.658 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191173106000001, 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:28.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191162341999998, 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:28.663 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.667 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191148206000001, 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.668 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:28.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191137675999997, 2024-09-11T05:53:28.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:28.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:28.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191125995, 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.677 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:28.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:28.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.191115676000003, 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:28.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:28.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:28.686 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.187602310000003, 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:28.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.187590311000001, 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:28.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:28.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.187573937000003, 2024-09-11T05:53:28.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:28.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:28.699 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.187560658999999, 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.700 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:28.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.704 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 55.187546765, 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:28.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:28.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 45.168023488000003, 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:28.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:28.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:28.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.714 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 45.156798858000002, 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:28.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:28.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:28.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:28.720 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:28.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:28.722 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:28.720+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:28.764 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.7 to 0.975979 (f9d9) 2024-09-11T05:53:28.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T05:53:28.849 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:28.846+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:170601 3.14 3:2859746d:::rbd_header.11249b359231:head [notify cookie 140146796152480 in=38b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:28.871499+0000", 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00019694299999999999, 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026132, 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:28.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 170601 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871499+0000", 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:28.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871499+0000", 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871501+0000", 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871510+0000", 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0680000000000003e-06 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871531+0000", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596999999999999e-05 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871542+0000", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1697e-05 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871652+0000", 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109678 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871759+0000", 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106895 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:170602 3.14 3.b62e9a14 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:28.871568+0000", 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00012786199999999999, 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040556099999999997, 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:28.876 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "queued for pg", 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 170602 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871568+0000", 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871568+0000", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871570+0000", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.243e-06 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871576+0000", 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1890000000000004e-06 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871594+0000", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7974000000000001e-05 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871607+0000", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3331e-05 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:28.871985+0000", 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037792899999999999 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:28.879 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:28.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:28.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:28.880 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-11T05:53:28.880 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-11T05:53:28.880 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:28.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:28.935 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:28.932+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:29.017 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:29.017 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:29.024 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:29.020+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:29.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:29.125 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-11T05:53:29.122+0000 7f26d8372640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26496) UID: 0 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.589451979000003, 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.199 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:53:29.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:53:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:53:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.204 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.589344075, 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:53:29.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.589285756000002, 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.210 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:53:29.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.589231634000001, 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:53:29.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:53:29.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588241375999999, 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:53:29.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588193226999998, 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:53:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588160961, 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:53:29.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:53:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:53:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588130948, 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588102016000001, 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.588060609999999, 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:53:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:53:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:53:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:53:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584414191, 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584331382999999, 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584254381000001, 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:53:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:53:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584182892999998, 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:53:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584126931, 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:53:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584087685, 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:53:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.584014336999999, 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.583984590999997, 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:53:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.580097373999998, 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:53:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.580042986000002, 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.292 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:29.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:29.294 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:29.290+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:29.394 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:29.388+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:29.420 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:29.420 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:29.420 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:29.420 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:29.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:29.522 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:29.520+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:29.596 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:29.592+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:29.597 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:29.597 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:29.597 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:29.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:29.699 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:29.696+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:29.781 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:29.781 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262161910000003, 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.782 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262129736000006, 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.787 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262106716000005, 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:53:29.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:53:29.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262070231999999, 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:53:29.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:53:29.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262042961000006, 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:53:29.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.262019933999994, 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:53:29.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261996218999997, 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.812 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:53:29.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261973404000003, 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.820 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261950444999997, 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:53:29.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:53:29.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.26192752, 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:53:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261903806000007, 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:53:29.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:29.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.26187711, 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:53:29.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261850744, 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:53:29.839 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261816049000004, 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:53:29.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.261636761000005, 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:53:29.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:53:29.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.853 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.25898608, 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.854 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:29.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:53:29.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:29.858 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.258964030000001, 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:53:29.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.863 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.258750903999996, 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:53:29.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.868 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.258720741000005, 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.869 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:53:29.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:29.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.258696843999999, 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:53:29.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:53:29.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:29.878 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:29.880 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is set 2024-09-11T05:53:29.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:29.881 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:29.876+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:29.980 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:29.976+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:30.053 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:30.054 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:30.054 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:30.054 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:30.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:30.111 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:30.108+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:30.182 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:30.180+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:30.235 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:30.235 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:30.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:30.283 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:30.278+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:30.383 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:30.380+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:30.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T05:53:30.416 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:30.416 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:30.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:30.416 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:30.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.209497188, 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:53:30.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.91031812999999, 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.421 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:53:30.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:30.425 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.027748596999999, 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:53:30.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.027145924999999, 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:30.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:53:30.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.026198940999997, 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:53:30.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:30.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.023445836999997, 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:30.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.800157458999998, 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.731407222000001, 2024-09-11T05:53:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:53:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.662497962000003, 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.022092143999998, 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:53:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:53:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:53:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.919244954, 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:53:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.918536187999997, 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:53:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:53:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.881953678999999, 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:53:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:53:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:166928 3.10 3:0eb31573:::rbd_header.112419b96064:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.885807591999999, 2024-09-11T05:53:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014429618, 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 166928 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527522+0000", 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.033e-06 2024-09-11T05:53:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527534+0000", 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1440000000000001e-05 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527573+0000", 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9672000000000002e-05 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527595+0000", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1342999999999999e-05 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527788+0000", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000193718 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527806+0000", 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7336000000000001e-05 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.532131+0000", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043253479999999997 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.532400+0000", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026928999999999999 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.533687+0000", 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001287016 2024-09-11T05:53:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541750+0000", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080628839999999993 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541782+0000", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1219e-05 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541870+0000", 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8473999999999994e-05 2024-09-11T05:53:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541949+0000", 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8844999999999995e-05 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3122 3.1d e127/83 3:b949eb71:::rbd_data.112419b96064.0000000000000074:1b v 127'265, mlcod=119'262)", 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.879193024999999, 2024-09-11T05:53:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094862829999999999, 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534135+0000", 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.607e-06 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534148+0000", 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2136e-05 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534172+0000", 2024-09-11T05:53:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4416000000000001e-05 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534192+0000", 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9779000000000002e-05 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534350+0000", 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157973 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534410+0000", 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0591000000000003e-05 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.543447+0000", 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090364490000000002 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.543620+0000", 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173332 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3123 3.1d e127/83 3:bd62314a:::rbd_data.112419b96064.000000000000000a:1b v 127'267, mlcod=119'262)", 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.877567644999999, 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088280100000000007, 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535761+0000", 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5030000000000001e-06 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535772+0000", 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1212999999999999e-05 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535795+0000", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3353000000000002e-05 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535814+0000", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8243000000000001e-05 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.539424+0000", 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036103630000000001 2024-09-11T05:53:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.539489+0000", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4951999999999997e-05 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.544500+0000", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050110959999999996 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.544587+0000", 2024-09-11T05:53:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7287000000000006e-05 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:166934 3.10 3:0eb31573:::rbd_header.112419b96064:head [watch unwatch cookie 140146796456272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.857841458999999, 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009557701, 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 166934 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555487+0000", 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555496+0000", 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.524e-06 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555518+0000", 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2393000000000001e-05 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555535+0000", 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7088999999999999e-05 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.556380+0000", 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00084499399999999995 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.556622+0000", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024143999999999999 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.557013+0000", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039166 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.561961+0000", 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049480169999999999 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.564779+0000", 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028181690000000001 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.564871+0000", 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1836999999999995e-05 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.565043+0000", 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171872 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:167482 3.10 3:0ee748d8:::rbd_data.11242d2cfbf4.0000000000000009:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.597987890999999, 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011179168, 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 167482 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815343+0000", 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0400000000000003e-06 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815363+0000", 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9933e-05 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815406+0000", 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3123e-05 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815434+0000", 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7659000000000001e-05 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815579+0000", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014556700000000001 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815591+0000", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1609e-05 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.822363+0000", 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067724400000000002 2024-09-11T05:53:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.822802+0000", 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043906199999999999 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.823130+0000", 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032797600000000001 2024-09-11T05:53:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.825983+0000", 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028525949999999999 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826209+0000", 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022627500000000001 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826362+0000", 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015248 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826518+0000", 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015640900000000001 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:167964 3.d e136/83 3:b6a339dc:::rbd_data.1124fc73d6eb.000000000000000a:head v 136'524, mlcod=135'522)", 2024-09-11T05:53:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.596754337, 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012102942, 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7700000000000003e-07 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816585+0000", 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1873e-05 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.819016+0000", 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024302849999999999 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.819063+0000", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7652999999999999e-05 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.822644+0000", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035804180000000001 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.822699+0000", 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4892000000000002e-05 2024-09-11T05:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.828556+0000", 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058579269999999998 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.828675+0000", 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119017 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3293 3.a e138/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:13 v 138'711, mlcod=137'708)", 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.698396582000001, 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010238209999999999, 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714931+0000", 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714938+0000", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.481e-06 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714968+0000", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9456000000000001e-05 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714981+0000", 2024-09-11T05:53:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3413e-05 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.715073+0000", 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1464000000000004e-05 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.715103+0000", 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0536e-05 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.725095+0000", 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099921300000000001 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.725168+0000", 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3264999999999997e-05 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:30.518 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:30.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:30.520 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:30.516+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:30.568 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:30.569 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:30.587 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:30.587 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:30.606 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:30.606 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:30.620 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:30.618+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:30.626 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:30.626 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:30.646 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:30.646 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:30.668 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:30.668 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:30.688 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:30.688 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:30.697 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:30.697 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:30.697 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:30.697 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:30.709 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:30.709 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:30.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:30.721 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:30.718+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:30.822 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-09-11T05:53:30.820+0000 7f36b8cba640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26523) UID: 0 2024-09-11T05:53:30.836 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:30.836 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:30.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:30.938 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:30.936+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:31.019 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.020 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.024 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:31.018+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:31.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T05:53:31.125 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:31.124+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:31.168 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.169 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T05:53:31.226 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:31.224+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:31.304 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.304 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T05:53:31.327 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:31.324+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:31.428 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:31.422+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:31.441 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.441 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T05:53:31.529 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:31.524+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:31.629 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:31.626+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:31.641 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.641 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T05:53:31.731 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:31.728+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:31.829 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:31.830 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:31.832 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:31.826+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:31.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T05:53:31.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T05:53:31.937 INFO:tasks.ceph.osd.1.smithi165.stderr:2024-09-11T05:53:31.934+0000 7f915f147640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26497) UID: 0 2024-09-11T05:53:32.013 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.013 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T05:53:32.077 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:32.074+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:32.135 INFO:tasks.ceph.osd.0.smithi165.stderr:2024-09-11T05:53:32.130+0000 7f0cd8db5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26491) UID: 0 2024-09-11T05:53:32.190 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.190 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T05:53:32.236 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:32.232+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:32.337 INFO:tasks.ceph.osd.2.smithi165.stderr:2024-09-11T05:53:32.334+0000 7fa271acd640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26493) UID: 0 2024-09-11T05:53:32.370 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.370 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T05:53:32.438 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-09-11T05:53:32.432+0000 7f1d3cc2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26528) UID: 0 2024-09-11T05:53:32.538 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is set 2024-09-11T05:53:32.539 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-09-11T05:53:32.536+0000 7f50ec06c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26522) UID: 0 2024-09-11T05:53:32.567 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.567 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T05:53:32.640 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-09-11T05:53:32.636+0000 7fa671087640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26525) UID: 0 2024-09-11T05:53:32.681 INFO:teuthology.orchestra.run.smithi165.stdout:All operations completed A-OK! 2024-09-11T05:53:32.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T05:53:32.730 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:32.730 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-09-11T05:53:32.740 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-11T05:53:32.761 INFO:tasks.thrashosds:joining thrashosds 2024-09-11T05:53:32.762 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.762 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T05:53:32.906 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0: {} 2024-09-11T05:53:32.906 INFO:teuthology.orchestra.run.smithi165.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T05:53:32.922 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1: {} 2024-09-11T05:53:32.923 INFO:teuthology.orchestra.run.smithi165.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T05:53:32.934 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-11T05:53:32.935 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T05:53:32.940 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2: {} 2024-09-11T05:53:32.941 INFO:teuthology.orchestra.run.smithi165.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T05:53:32.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T05:53:32.958 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3: {} 2024-09-11T05:53:32.958 INFO:teuthology.orchestra.run.smithi165.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T05:53:32.978 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4: {} 2024-09-11T05:53:32.978 INFO:teuthology.orchestra.run.smithi165.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T05:53:32.999 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5: {} 2024-09-11T05:53:33.000 INFO:teuthology.orchestra.run.smithi165.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T05:53:33.020 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6: {} 2024-09-11T05:53:33.020 INFO:teuthology.orchestra.run.smithi165.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-11T05:53:33.041 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7: {} 2024-09-11T05:53:33.041 INFO:teuthology.orchestra.run.smithi165.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-11T05:53:33.067 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:33.067 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-11T05:53:33.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:78242 3.8 3:169d4227:::rbd_data.1124c865af2a.0000000000000006:head [read 5448295~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 108.22338432399999, 2024-09-11T05:53:33.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0007043529999999, 2024-09-11T05:53:33.069 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.069 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.069 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.069 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.070 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.070 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78242 2024-09-11T05:53:33.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.070 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:33.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841641+0000", 2024-09-11T05:53:33.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841644+0000", 2024-09-11T05:53:33.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5229999999999998e-06 2024-09-11T05:53:33.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841656+0000", 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841679+0000", 2024-09-11T05:53:33.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3337e-05 2024-09-11T05:53:33.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841690+0000", 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0533000000000001e-05 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841764+0000", 2024-09-11T05:53:33.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4085000000000006e-05 2024-09-11T05:53:33.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:33.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:44.841773+0000", 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841482+0000", 2024-09-11T05:53:33.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99970911299999998 2024-09-11T05:53:33.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.841947+0000", 2024-09-11T05:53:33.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464296 2024-09-11T05:53:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:51:45.842346+0000", 2024-09-11T05:53:33.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039916600000000001 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.081 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126415 3.10 3:0b7b6f02:::rbd_id.image_client.0-clone30:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-11T05:53:33.082 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:33.082 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.490767867000002, 2024-09-11T05:53:33.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96907722399999996, 2024-09-11T05:53:33.082 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.082 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126415 2024-09-11T05:53:33.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.084 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:33.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574258+0000", 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574259+0000", 2024-09-11T05:53:33.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1399999999999995e-07 2024-09-11T05:53:33.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574264+0000", 2024-09-11T05:53:33.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2530000000000004e-06 2024-09-11T05:53:33.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574276+0000", 2024-09-11T05:53:33.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2067e-05 2024-09-11T05:53:33.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574287+0000", 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0400999999999999e-05 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574342+0000", 2024-09-11T05:53:33.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4956e-05 2024-09-11T05:53:33.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:33.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:22.574348+0000", 2024-09-11T05:53:33.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-09-11T05:53:33.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538120+0000", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.963771453 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538293+0000", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017369799999999999 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.538544+0000", 2024-09-11T05:53:33.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025070399999999998 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.542804+0000", 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042594759999999999 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543211+0000", 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000406974 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543278+0000", 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7593000000000001e-05 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.543335+0000", 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6954999999999999e-05 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:126477 3.17 3:e86963fc:::rbd_object_map.11243ca4538.0000000000000020:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-11T05:53:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.336441995000001, 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81187869300000004, 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 126477 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728584+0000", 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728585+0000", 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1699999999999997e-07 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728589+0000", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1110000000000001e-06 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728599+0000", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0451e-05 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728604+0000", 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9520000000000002e-06 2024-09-11T05:53:33.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728652+0000", 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7596000000000003e-05 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:23.728658+0000", 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.539864+0000", 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.81120572800000001 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540100+0000", 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023574300000000001 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:24.540462+0000", 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036269600000000003 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.099 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133883 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 2985984~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635506900999999, 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070574868999999998, 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133883 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429519+0000", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2499999999999999e-07 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429520+0000", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429524+0000", 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-11T05:53:33.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429526+0000", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5450000000000002e-06 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499909+0000", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070382396 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.499950+0000", 2024-09-11T05:53:33.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1767000000000002e-05 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500094+0000", 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000143409 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133884 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3047424~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:33.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635496250999999, 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070913033, 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133884 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429530+0000", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1799999999999999e-07 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429531+0000", 2024-09-11T05:53:33.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.136e-06 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429534+0000", 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4309999999999998e-06 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429537+0000", 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.615e-06 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500153+0000", 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070615935000000005 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500203+0000", 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0404000000000001e-05 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500443+0000", 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000239294 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.108 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133885 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3108864~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635485611999997, 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071210712999999995, 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133885 2024-09-11T05:53:33.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429540+0000", 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429541+0000", 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3799999999999998e-07 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429542+0000", 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1480000000000001e-06 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429545+0000", 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3670000000000001e-06 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429548+0000", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6680000000000001e-06 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500491+0000", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070943299000000001 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500540+0000", 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9330999999999999e-05 2024-09-11T05:53:33.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500751+0000", 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021056199999999999 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133886 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3170304~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635473429999998, 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071529447999999995, 2024-09-11T05:53:33.113 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133886 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429552+0000", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429553+0000", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.17e-07 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429554+0000", 2024-09-11T05:53:33.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429557+0000", 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5080000000000001e-06 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429560+0000", 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5179999999999999e-06 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500791+0000", 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071231118999999996 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.500861+0000", 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0028000000000001e-05 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501082+0000", 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022086299999999999 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133887 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3231744~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635462457000003, 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071793621000000002, 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.118 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133887 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429563+0000", 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429564+0000", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429565+0000", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-06 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429568+0000", 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4010000000000001e-06 2024-09-11T05:53:33.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429571+0000", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5059999999999998e-06 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501119+0000", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071548226000000006 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501161+0000", 2024-09-11T05:53:33.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1819000000000001e-05 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501357+0000", 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019621300000000001 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133888 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3293184~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:33.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635451693, 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071978639999999997, 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133888 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429574+0000", 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2100000000000001e-07 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429578+0000", 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0890000000000002e-06 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429582+0000", 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6219999999999998e-06 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429585+0000", 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.621e-06 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501397+0000", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071812200000000007 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501468+0000", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1445000000000005e-05 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501553+0000", 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4442000000000002e-05 2024-09-11T05:53:33.126 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133889 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3354624~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635437557000003, 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072317797000000003, 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.127 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133889 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429588+0000", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3300000000000001e-07 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429590+0000", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.158e-06 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429593+0000", 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.281e-06 2024-09-11T05:53:33.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429595+0000", 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5670000000000002e-06 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501588+0000", 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071992267999999998 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501661+0000", 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2848000000000005e-05 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501906+0000", 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024544200000000001 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133890 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3416064~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:33.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635427026999999, 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072518831000000006, 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133890 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429599+0000", 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.11e-07 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429600+0000", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-06 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429603+0000", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429607+0000", 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6030000000000001e-06 2024-09-11T05:53:33.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.501954+0000", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07234699 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502028+0000", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4109999999999993e-05 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502118+0000", 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9487000000000006e-05 2024-09-11T05:53:33.135 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133891 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3477504~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635415346000002, 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072717460999999997, 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.136 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133891 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429610+0000", 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429611+0000", 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.16e-07 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429612+0000", 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.113e-06 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429615+0000", 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2569999999999999e-06 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429617+0000", 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4660000000000002e-06 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502154+0000", 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072536446000000004 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502227+0000", 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3498999999999995e-05 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502328+0000", 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010046400000000001 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133892 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3538944~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.140 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.635405026999997, 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.072898334999999995, 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133892 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.429621+0000", 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433016+0000", 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033953870000000001 2024-09-11T05:53:33.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433021+0000", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433036+0000", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4687e-05 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433042+0000", 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6189999999999999e-06 2024-09-11T05:53:33.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502361+0000", 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.069319066999999998 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502431+0000", 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0557999999999998e-05 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.502519+0000", 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7705000000000001e-05 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133898 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3907584~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.631891660999997, 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070832433, 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133898 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433134+0000", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433136+0000", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6649999999999999e-06 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433140+0000", 2024-09-11T05:53:33.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8489999999999999e-06 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433143+0000", 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7949999999999999e-06 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503691+0000", 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070548134999999998 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503763+0000", 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.216e-05 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.503967+0000", 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020358900000000001 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133899 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 3969024~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.631879662000003, 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071085148000000001, 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133899 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:33.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433146+0000", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4699999999999998e-07 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433148+0000", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2359999999999999e-06 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433153+0000", 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9660000000000004e-06 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433155+0000", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8509999999999999e-06 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504003+0000", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070847183999999994 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504064+0000", 2024-09-11T05:53:33.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0930000000000001e-05 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504231+0000", 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000167734 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133900 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4030464~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.631863287999998, 2024-09-11T05:53:33.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071273000000000003, 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133900 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:33.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433162+0000", 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433163+0000", 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.28e-07 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433164+0000", 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433169+0000", 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8029999999999996e-06 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433172+0000", 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9069999999999999e-06 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504270+0000", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071098061000000004 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504345+0000", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5136999999999996e-05 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504435+0000", 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0391999999999998e-05 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133901 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4091904~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.631850010000001, 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071563782000000006, 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133901 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433176+0000", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1899999999999999e-07 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433177+0000", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2130000000000001e-06 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433181+0000", 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5049999999999999e-06 2024-09-11T05:53:33.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433183+0000", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.689e-06 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504472+0000", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.07128893 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504521+0000", 2024-09-11T05:53:33.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8773999999999998e-05 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504740+0000", 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218452 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:133902 3.b 3:d4522013:::rbd_data.1124fd5e4ce2.0000000000000000:24 [read 4153344~4096] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e96)", 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:33.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.631836116000002, 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071754462000000005, 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133902 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:33.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433190+0000", 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.35e-07 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433191+0000", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.22e-06 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433195+0000", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.551e-06 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.433198+0000", 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.841e-06 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504784+0000", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071586424999999995 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504858+0000", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3736e-05 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:33.504944+0000", 2024-09-11T05:53:33.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6453999999999996e-05 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161532 3.1 3:81ea9700:::rbd_data.11243956a123.000000000000000e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.612312838999998, 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.103148505, 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161532 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452713+0000", 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452716+0000", 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0599999999999999e-06 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452728+0000", 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2469e-05 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452756+0000", 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8019000000000001e-05 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.452777+0000", 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0740000000000001e-05 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551402+0000", 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.098624751999999996 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.551667+0000", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026531800000000001 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.552042+0000", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555281+0000", 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032392100000000002 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555329+0000", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7672e-05 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555499+0000", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017037399999999999 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.555861+0000", 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036230399999999998 2024-09-11T05:53:33.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.4388.0:161534 3.1 3:80d90b55:::rbd_data.11243956a123.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.601088208999997, 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.093582095000000004, 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.4388", 2024-09-11T05:53:33.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161534 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463938+0000", 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463940+0000", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463950+0000", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0823e-05 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463977+0000", 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6849e-05 2024-09-11T05:53:33.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.463992+0000", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4379000000000001e-05 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553048+0000", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089056249000000004 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.553408+0000", 2024-09-11T05:53:33.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036001400000000003 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.554156+0000", 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074806700000000005 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557191+0000", 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030349069999999999 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:33.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557271+0000", 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.996e-05 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557384+0000", 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112726 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-11T05:52:43.557520+0000", 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013611100000000001 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-11T05:53:33.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-11T05:53:33.181 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:33.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T05:53:33.313 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:33.313 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:33.314 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:33.314 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:33.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T05:53:33.445 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:33.445 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:33.445 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:33.445 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:33.445 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:33.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.581 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128648 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 7405568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.970995450999993, 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083912234000000002, 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128648 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606508+0000", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606510+0000", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.37e-06 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606515+0000", 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7600000000000002e-06 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606530+0000", 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5116000000000001e-05 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606542+0000", 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2266000000000001e-05 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690155+0000", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083613442999999996 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690252+0000", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6354999999999995e-05 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690421+0000", 2024-09-11T05:53:33.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016892400000000001 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128649 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 8978432~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.970887547000004, 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084170197000000002, 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128649 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606616+0000", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606618+0000", 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5370000000000001e-06 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606624+0000", 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7790000000000001e-06 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606644+0000", 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0883999999999999e-05 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606658+0000", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3942e-05 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690508+0000", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083849897000000007 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690622+0000", 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113695 2024-09-11T05:53:33.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690786+0000", 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000164463 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128650 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 10813440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.970829227999999, 2024-09-11T05:53:33.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084503304000000001, 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128650 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:33.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606675+0000", 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606676+0000", 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.223e-06 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606681+0000", 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9699999999999998e-06 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606699+0000", 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8672999999999999e-05 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606714+0000", 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4389000000000001e-05 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690871+0000", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084157264999999995 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.690996+0000", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012447399999999999 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691178+0000", 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018231000000000001 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128651 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 12386304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.970775106000005, 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084777446000000006, 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128651 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.606729+0000", 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607561+0000", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083258900000000001 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607589+0000", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7895999999999999e-05 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607635+0000", 2024-09-11T05:53:33.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5392000000000002e-05 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607654+0000", 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9539999999999999e-05 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691229+0000", 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083575208999999998 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691339+0000", 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109362 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691506+0000", 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016745799999999999 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128652 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7733248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969784848000003, 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084122225999999994, 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128652 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607719+0000", 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607720+0000", 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.184e-06 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607726+0000", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6500000000000001e-06 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607739+0000", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3655999999999999e-05 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607753+0000", 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3485999999999999e-05 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691572+0000", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083819299999999999 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691709+0000", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136723 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691841+0000", 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013222700000000001 2024-09-11T05:53:33.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128653 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9568256~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969736698999995, 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084438838000000002, 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128653 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607767+0000", 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607768+0000", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8400000000000003e-07 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607772+0000", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6459999999999999e-06 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607781+0000", 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6430000000000007e-06 2024-09-11T05:53:33.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607790+0000", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3499999999999997e-06 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691902+0000", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084112362999999996 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692029+0000", 2024-09-11T05:53:33.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126871 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692206+0000", 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177181 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128654 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:33.608 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969704433000004, 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084798392, 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128654 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:33.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607799+0000", 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607800+0000", 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5000000000000001e-07 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607804+0000", 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1669999999999999e-06 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607812+0000", 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8629999999999997e-06 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607820+0000", 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7759999999999997e-06 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692393+0000", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084573111000000006 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692499+0000", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105703 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692598+0000", 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8821999999999997e-05 2024-09-11T05:53:33.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128655 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 12976128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969674420000004, 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085094580000000003, 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.613 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128655 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607829+0000", 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607830+0000", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2600000000000002e-07 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607833+0000", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0900000000000001e-06 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607842+0000", 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7169999999999997e-06 2024-09-11T05:53:33.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607849+0000", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3259999999999998e-06 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692662+0000", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084812289999999999 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692789+0000", 2024-09-11T05:53:33.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012718399999999999 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692924+0000", 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013524699999999999 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128656 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 7897088~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969645487999998, 2024-09-11T05:53:33.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085450656999999999, 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128656 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607858+0000", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607859+0000", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8700000000000005e-07 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607873+0000", 2024-09-11T05:53:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3739e-05 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607882+0000", 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3670000000000003e-06 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607890+0000", 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.692998+0000", 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085108236000000004 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693147+0000", 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014830400000000001 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693309+0000", 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016247399999999999 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128657 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.969604082000004, 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085730549000000003, 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128657 2024-09-11T05:53:33.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.607900+0000", 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611106+0000", 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032058170000000001 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611119+0000", 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3276e-05 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611152+0000", 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2842999999999997e-05 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611172+0000", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0199000000000001e-05 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693378+0000", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082206204000000005 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693507+0000", 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012928700000000001 2024-09-11T05:53:33.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.693630+0000", 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012292299999999999 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128663 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13303808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965957662999998, 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084022066000000006, 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128663 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611546+0000", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611548+0000", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.53e-06 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611554+0000", 2024-09-11T05:53:33.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611578+0000", 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3502999999999998e-05 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611608+0000", 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0032e-05 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695317+0000", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083709190000000003 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695444+0000", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012714599999999999 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695568+0000", 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124325 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128664 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 8486912~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965874854999996, 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084275155000000004, 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128664 2024-09-11T05:53:33.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611629+0000", 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611630+0000", 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.387e-06 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611636+0000", 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9669999999999998e-06 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611657+0000", 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0505000000000001e-05 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611672+0000", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4963999999999999e-05 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695632+0000", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083960145999999999 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695754+0000", 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121983 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695904+0000", 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015020300000000001 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128665 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 10059776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965797852999998, 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084418035000000002, 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128665 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611706+0000", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611707+0000", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2699999999999999e-06 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611715+0000", 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.943e-06 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611742+0000", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6712000000000001e-05 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611757+0000", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4973999999999999e-05 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.695961+0000", 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084203730000000004 2024-09-11T05:53:33.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696051+0000", 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0280999999999998e-05 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696124+0000", 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3125000000000002e-05 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128666 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 11894784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965726364999995, 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084640114000000002, 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128666 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611777+0000", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611778+0000", 2024-09-11T05:53:33.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.02e-06 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611784+0000", 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1529999999999996e-06 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611803+0000", 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9536000000000001e-05 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611811+0000", 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1200000000000002e-06 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696181+0000", 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084369361000000004 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696302+0000", 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012158 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696418+0000", 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115344 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128667 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 13467648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965670403000004, 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084760901999999999, 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.644 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128667 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611833+0000", 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611834+0000", 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611838+0000", 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6500000000000002e-06 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611853+0000", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4888999999999999e-05 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611861+0000", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0630000000000006e-06 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696461+0000", 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084600578999999995 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696522+0000", 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0853000000000002e-05 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696594+0000", 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2157999999999996e-05 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128668 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965631157000004, 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085049161999999998, 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128668 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611873+0000", 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611913+0000", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9478999999999999e-05 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611925+0000", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2027e-05 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611934+0000", 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7390000000000005e-06 2024-09-11T05:53:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696646+0000", 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084712600999999998 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696768+0000", 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122223 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696922+0000", 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015345699999999999 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128669 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965557809000003, 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085187339000000001, 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128669 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611946+0000", 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611947+0000", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611950+0000", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3730000000000001e-06 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611959+0000", 2024-09-11T05:53:33.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611967+0000", 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6969999999999993e-06 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.696970+0000", 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085003521999999998 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697033+0000", 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2682999999999996e-05 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697133+0000", 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100581 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128670 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 3866624~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.965528062999994, 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085480836000000004, 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128670 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.611976+0000", 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615435+0000", 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034590480000000002 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615448+0000", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3171e-05 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615472+0000", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4473000000000001e-05 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615507+0000", 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4103000000000002e-05 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697185+0000", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081678857999999993 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697306+0000", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012052700000000001 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.697457+0000", 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015065599999999999 2024-09-11T05:53:33.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128678 3.1b 3:dfb66978:::rbd_data.1124cfe3363c.0000000000000001:21 [read 4521984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.961640845999995, 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083989020999999997, 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128678 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615863+0000", 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615864+0000", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9400000000000004e-07 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615868+0000", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3959999999999999e-06 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615891+0000", 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3244e-05 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615903+0000", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1151999999999999e-05 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699574+0000", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083671385000000001 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699720+0000", 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014611600000000001 2024-09-11T05:53:33.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699852+0000", 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131934 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:128679 3.1b 3:d8efb116:::rbd_data.1124cfe3363c.0000000000000000:21 [read 6356992~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.961586457999999, 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084085595999999999, 2024-09-11T05:53:33.666 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 128679 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615917+0000", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619069+0000", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003151933 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619079+0000", 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0112000000000001e-05 2024-09-11T05:53:33.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619102+0000", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2303e-05 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.619116+0000", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4626e-05 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699871+0000", 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080754630999999993 2024-09-11T05:53:33.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.699926+0000", 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5469e-05 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.700003+0000", 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6521999999999997e-05 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:33.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:33.671 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:33.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T05:53:33.800 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:33.800 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:33.800 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:33.801 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-11T05:53:33.802 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-11T05:53:33.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-09-11T05:53:33.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T05:53:33.936 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:33.936 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:33.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T05:53:34.072 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:34.072 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113460 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 251904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552990907999998, 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064261636999999996, 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113460 2024-09-11T05:53:34.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516073+0000", 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516074+0000", 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5199999999999997e-07 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516082+0000", 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4600000000000003e-06 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516090+0000", 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1380000000000007e-06 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516097+0000", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580141+0000", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064043640999999998 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580237+0000", 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6700999999999995e-05 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580335+0000", 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7306000000000007e-05 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113461 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 266240~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552958734000001, 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424816999999995, 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.077 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113461 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516105+0000", 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516106+0000", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2399999999999998e-07 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516108+0000", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4370000000000001e-06 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516115+0000", 2024-09-11T05:53:34.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9820000000000002e-06 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516121+0000", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7100000000000004e-06 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580370+0000", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064248964000000006 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580444+0000", 2024-09-11T05:53:34.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4238000000000003e-05 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580530+0000", 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5962000000000001e-05 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113462 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 276480~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552935714, 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064532859999999997, 2024-09-11T05:53:34.081 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113462 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516128+0000", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516129+0000", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.01e-07 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516131+0000", 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2419999999999999e-06 2024-09-11T05:53:34.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516138+0000", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0180000000000004e-06 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516143+0000", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4770000000000002e-06 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580557+0000", 2024-09-11T05:53:34.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064413735999999999 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580600+0000", 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3316000000000003e-05 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580661+0000", 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.067e-05 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113463 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 286720~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552899229999994, 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064750685000000002, 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113463 2024-09-11T05:53:34.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516165+0000", 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1799999999999995e-07 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516168+0000", 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9979999999999999e-06 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516177+0000", 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0950000000000002e-06 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516184+0000", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3040000000000001e-06 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580714+0000", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064530847000000002 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580811+0000", 2024-09-11T05:53:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6319999999999999e-05 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580915+0000", 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104603 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113464 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 296960~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552871959000001, 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064963611000000004, 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113464 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516192+0000", 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000004e-07 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516195+0000", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5210000000000001e-06 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516202+0000", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3220000000000004e-06 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516208+0000", 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5079999999999996e-06 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.580948+0000", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064740353 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581035+0000", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6689999999999998e-05 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581156+0000", 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120695 2024-09-11T05:53:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113465 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 307200~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552848932000003, 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065176033999999994, 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113465 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516215+0000", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1399999999999997e-07 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516218+0000", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8940000000000002e-06 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516226+0000", 2024-09-11T05:53:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3849999999999996e-06 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516231+0000", 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6119999999999998e-06 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581200+0000", 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064968387000000002 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581291+0000", 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1688999999999996e-05 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581391+0000", 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9553e-05 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113466 3.5 3:a7662bc7:::rbd_data.112419b96064.000000000000000b:1b [read 317440~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552825217000006, 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065378547999999995, 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113466 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516239+0000", 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516242+0000", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.52e-06 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516249+0000", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0129999999999996e-06 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516254+0000", 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4389999999999999e-06 2024-09-11T05:53:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581424+0000", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065170259999999994 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581513+0000", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8928e-05 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581617+0000", 2024-09-11T05:53:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103884 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113506 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1178624~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552802401999998, 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065588832, 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113506 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:34.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516262+0000", 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0399999999999996e-07 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516264+0000", 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4509999999999999e-06 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516272+0000", 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3080000000000002e-06 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516278+0000", 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7440000000000004e-06 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581677+0000", 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065399955999999995 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581750+0000", 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2575000000000002e-05 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581850+0000", 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100294 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113507 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 861184~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552779443000006, 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065562915999999999, 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113507 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516284+0000", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516285+0000", 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7599999999999997e-07 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516287+0000", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2919999999999998e-06 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516294+0000", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7889999999999997e-06 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516300+0000", 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8660000000000002e-06 2024-09-11T05:53:34.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581647+0000", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065347304999999994 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581744+0000", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6744999999999996e-05 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581847+0000", 2024-09-11T05:53:34.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344300000000001 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113508 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 871424~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552756517999995, 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065731366999999999, 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113508 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516307+0000", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516308+0000", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4700000000000002e-07 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516310+0000", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1509999999999998e-06 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516317+0000", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0609999999999998e-06 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516323+0000", 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9710000000000001e-06 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581887+0000", 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065564007999999993 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.581970+0000", 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2931999999999998e-05 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582039+0000", 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8796999999999999e-05 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113509 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 881664~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552732804000001, 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065853796000000006, 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113509 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516331+0000", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516332+0000", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0100000000000005e-07 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516334+0000", 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6220000000000001e-06 2024-09-11T05:53:34.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516344+0000", 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5070000000000006e-06 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516350+0000", 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8050000000000003e-06 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582069+0000", 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065719412000000005 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582125+0000", 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6014999999999998e-05 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582185+0000", 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9933999999999999e-05 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.121 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113510 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 891904~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552706107999995, 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065985614999999997, 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113510 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516358+0000", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2099999999999997e-07 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516361+0000", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6249999999999999e-06 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516371+0000", 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6749999999999997e-06 2024-09-11T05:53:34.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516376+0000", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8320000000000002e-06 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582229+0000", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065852624999999998 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582284+0000", 2024-09-11T05:53:34.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5186999999999999e-05 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582343+0000", 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9150000000000001e-05 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113511 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 902144~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552679741999995, 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066158033000000005, 2024-09-11T05:53:34.126 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113511 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516384+0000", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516385+0000", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0299999999999999e-07 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516393+0000", 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516404+0000", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1318000000000001e-05 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516410+0000", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999999e-06 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582381+0000", 2024-09-11T05:53:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065970881999999995 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582453+0000", 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2212000000000006e-05 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9011000000000005e-05 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113512 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 1168384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552645046999999, 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066123322999999998, 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113512 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516419+0000", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516563+0000", 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014427999999999999 2024-09-11T05:53:34.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516567+0000", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9870000000000001e-06 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516578+0000", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1003e-05 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516586+0000", 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-06 2024-09-11T05:53:34.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582371+0000", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.065785165000000007 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582448+0000", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.648e-05 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582542+0000", 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4687999999999999e-05 2024-09-11T05:53:34.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113513 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 912384~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.552465759, 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066126580000000004, 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.135 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113513 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.516598+0000", 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519107+0000", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025087349999999998 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519113+0000", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2770000000000002e-06 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519126+0000", 2024-09-11T05:53:34.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3307e-05 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519135+0000", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7569999999999993e-06 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582566+0000", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063431133000000001 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582643+0000", 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6174999999999995e-05 2024-09-11T05:53:34.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.582725+0000", 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2195999999999997e-05 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113524 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 760832~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.549815077999995, 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064255926000000005, 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.139 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113524 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519249+0000", 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7100000000000002e-07 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519252+0000", 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5330000000000001e-06 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519259+0000", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519264+0000", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1649999999999997e-06 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583303+0000", 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064038827000000006 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583393+0000", 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0476000000000006e-05 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583505+0000", 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111743 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113525 3.5 3:a2acbf20:::rbd_data.11249bd3c27a.0000000000000000:1a [read 781312~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.549793027999996, 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064424618000000003, 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113525 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519271+0000", 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0699999999999998e-07 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519274+0000", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4540000000000001e-06 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519455+0000", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018120000000000001 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519466+0000", 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0900000000000001e-05 2024-09-11T05:53:34.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583545+0000", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064078670000000004 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583608+0000", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3536999999999998e-05 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583696+0000", 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7449999999999998e-05 2024-09-11T05:53:34.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113532 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1188864~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.549579902000005, 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064175549999999998, 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.148 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113532 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519484+0000", 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519485+0000", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0999999999999998e-07 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519489+0000", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3229999999999999e-06 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519499+0000", 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7569999999999999e-06 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519505+0000", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3439999999999997e-06 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583537+0000", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064032045999999995 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583598+0000", 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1088999999999997e-05 2024-09-11T05:53:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583660+0000", 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1280999999999998e-05 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113533 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1199104~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.549549739, 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064352063000000001, 2024-09-11T05:53:34.152 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113533 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519514+0000", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519515+0000", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6800000000000001e-07 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519518+0000", 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1489999999999998e-06 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519525+0000", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8680000000000001e-06 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519530+0000", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5339999999999999e-06 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583682+0000", 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064151528999999999 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583761+0000", 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9427999999999999e-05 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583866+0000", 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010508700000000001 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.156 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:113534 3.1d 3:bf066ae8:::rbd_data.11249bd3c27a.0000000000000002:1a [read 1209344~1024] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.549525841999994, 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064470666999999995, 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113534 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.519538+0000", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520806+0000", 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012674940000000001 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520812+0000", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8959999999999997e-06 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520830+0000", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7125999999999998e-05 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.520841+0000", 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1756000000000001e-05 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583900+0000", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.063059120999999996 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.583944+0000", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4019e-05 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:04.584009+0000", 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4255000000000001e-05 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.161 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.161 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.161 INFO:tasks.ceph.mon.a.smithi165.stderr:2024-09-11T05:53:34.102+0000 7f2b6abd3640 -1 mon.a@0(leader).osd e154 definitely_dead 0 2024-09-11T05:53:34.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-11T05:53:34.289 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.289 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:34.289 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:34.289 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-11T05:53:34.428 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:34.428 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-11T05:53:34.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:61965 3.1e e52/23 3:7900cbed:::rbd_data.1124767f15a5.000000000000003a:head v 52'174, mlcod=51'172)", 2024-09-11T05:53:34.568 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.362964944, 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090058870000000006, 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203830+0000", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.203831+0000", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.429e-06 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204684+0000", 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085310500000000005 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204703+0000", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204714+0000", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.058e-05 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204820+0000", 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010611999999999999 2024-09-11T05:53:34.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.204838+0000", 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8345000000000001e-05 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212785+0000", 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079471090000000008 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:40.212836+0000", 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0204999999999998e-05 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:77867 3.a e52/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:head v 52'169, mlcod=52'166)", 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 113.06378588600001, 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014214889, 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.573 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503009+0000", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.503010+0000", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169999999999999e-06 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506582+0000", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035719189999999998 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506610+0000", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348e-05 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506622+0000", 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2234e-05 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506695+0000", 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2673e-05 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.506722+0000", 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7478e-05 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517158+0000", 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010436422000000001 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:51:41.517224+0000", 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5197999999999995e-05 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127916 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'696, mlcod=93'690)", 2024-09-11T05:53:34.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.181216352999996, 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098851760000000007, 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385578+0000", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385580+0000", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385588+0000", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5129999999999996e-06 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385612+0000", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3842e-05 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.385629+0000", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.713e-05 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386550+0000", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092045400000000002 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386600+0000", 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0136000000000002e-05 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395390+0000", 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087900419999999996 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395463+0000", 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3734999999999998e-05 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127917 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'697, mlcod=93'690)", 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.180613680999997, 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093827149999999998, 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386181+0000", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386183+0000", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0099999999999998e-06 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386197+0000", 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4277000000000001e-05 2024-09-11T05:53:34.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386225+0000", 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7521999999999998e-05 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.386243+0000", 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8156999999999999e-05 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387914+0000", 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001670845 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387949+0000", 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4909000000000001e-05 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395470+0000", 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007521837 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395564+0000", 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3158000000000005e-05 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127918 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'698, mlcod=93'692)", 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.179666697000002, 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085313179999999995, 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.586 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387128+0000", 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387130+0000", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1500000000000002e-06 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387146+0000", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6129000000000001e-05 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387181+0000", 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4619000000000002e-05 2024-09-11T05:53:34.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.387206+0000", 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5663000000000001e-05 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389293+0000", 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020865850000000002 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389351+0000", 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8415000000000002e-05 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395574+0000", 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062226060000000003 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395659+0000", 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5150999999999998e-05 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:127922 3.10 e93/89 3:084b6885:::rbd_object_map.11241be40657:head v 93'702, mlcod=93'692)", 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.176913592999995, 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093309499999999993, 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389881+0000", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389883+0000", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6869999999999999e-06 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389897+0000", 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4090000000000001e-05 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389942+0000", 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4999000000000001e-05 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.389959+0000", 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7445999999999999e-05 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395914+0000", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005955162 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.395965+0000", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1016000000000001e-05 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399095+0000", 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031295630000000001 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.399212+0000", 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116987 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128863 3.d e93/83 3:b1220982:::rbd_data.11241be40657.0000000000000001:head v 93'374, mlcod=93'372)", 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.953625215000002, 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011267495000000001, 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613169+0000", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.613172+0000", 2024-09-11T05:53:34.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1229999999999998e-06 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615767+0000", 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002595099 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615802+0000", 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5176000000000003e-05 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615825+0000", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.615948+0000", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123392 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.616001+0000", 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3112999999999998e-05 2024-09-11T05:53:34.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624326+0000", 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083250159999999993 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.624437+0000", 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110773 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128871 3.1d e93/83 3:bdf0ded5:::rbd_data.11241be40657.0000000000000005:head v 93'187, mlcod=93'185)", 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.884874977999999, 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093048829999999999, 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681920+0000", 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.681921+0000", 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.858e-06 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683016+0000", 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010949950000000001 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683042+0000", 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5714000000000001e-05 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683070+0000", 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.777e-05 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683169+0000", 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9495999999999997e-05 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.683210+0000", 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0262000000000002e-05 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691132+0000", 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079217950000000006 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.691225+0000", 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2992999999999995e-05 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:128882 3.3 e93/90 3:c6fc4ed5:::rbd_data.11241be40657.000000000000000a:head v 93'478, mlcod=93'476)", 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.815965718000001, 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091575519999999994, 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750829+0000", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.750832+0000", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9560000000000002e-06 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752693+0000", 2024-09-11T05:53:34.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018613080000000001 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752729+0000", 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6035000000000003e-05 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752745+0000", 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6189999999999999e-05 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752846+0000", 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100715 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.752882+0000", 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5907e-05 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759903+0000", 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00702149 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:27.759986+0000", 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2950999999999999e-05 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157014 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'418, mlcod=99'413)", 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.175559900000003, 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085960029999999996, 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391235+0000", 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391236+0000", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.714e-06 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391250+0000", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3959e-05 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391279+0000", 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9025000000000001e-05 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391297+0000", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7921999999999999e-05 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391369+0000", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1785000000000005e-05 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.391382+0000", 2024-09-11T05:53:34.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2697000000000001e-05 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399787+0000", 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084048249999999994 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.399831+0000", 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4076000000000003e-05 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.613 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157095 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'441, mlcod=99'435)", 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.072712709999998, 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095000039999999994, 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494082+0000", 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494083+0000", 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1699999999999998e-07 2024-09-11T05:53:34.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494087+0000", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.887e-06 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494100+0000", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2102e-05 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494107+0000", 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5730000000000002e-06 2024-09-11T05:53:34.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495245+0000", 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011376610000000001 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.495305+0000", 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9926999999999997e-05 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503458+0000", 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081536639999999997 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503582+0000", 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123573 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:157096 3.1e e99/23 3:79f3c817:::rbd_object_map.1124fa04ae41:head v 99'442, mlcod=99'436)", 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:34.618 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.072003944000002, 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088837399999999993, 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:34.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494791+0000", 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999996e-07 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494796+0000", 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494803+0000", 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8809999999999998e-06 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.494807+0000", 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4819999999999996e-06 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496408+0000", 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016010029999999999 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.496429+0000", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0332000000000001e-05 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503589+0000", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071602769999999996 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:38.503674+0000", 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5408000000000006e-05 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:161540 3.10 3:08571df6:::rbd_data.11243956a123.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 10240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.035421435000003, 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.255929992, 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 161540 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531373+0000", 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531375+0000", 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.641e-06 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531382+0000", 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7209999999999997e-06 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531392+0000", 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0747000000000001e-05 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531401+0000", 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6079999999999996e-06 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531475+0000", 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4145000000000002e-05 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.531486+0000", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1462e-05 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.783743+0000", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25225610399999998 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784165+0000", 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042198700000000002 2024-09-11T05:53:34.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.784400+0000", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023546999999999999 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.785923+0000", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001523017 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787113+0000", 2024-09-11T05:53:34.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011895219999999999 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787233+0000", 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119982 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:43.787303+0000", 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0586000000000005e-05 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:166928 3.10 3:0eb31573:::rbd_header.112419b96064:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.039275348, 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014429618, 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 166928 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.630 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527519+0000", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527522+0000", 2024-09-11T05:53:34.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.033e-06 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527534+0000", 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1440000000000001e-05 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527573+0000", 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9672000000000002e-05 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527595+0000", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1342999999999999e-05 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527788+0000", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000193718 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.527806+0000", 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7336000000000001e-05 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.532131+0000", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043253479999999997 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.532400+0000", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026928999999999999 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.533687+0000", 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001287016 2024-09-11T05:53:34.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541750+0000", 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080628839999999993 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541782+0000", 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1219e-05 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541870+0000", 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8473999999999994e-05 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.541949+0000", 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8844999999999995e-05 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3122 3.1d e127/83 3:b949eb71:::rbd_data.112419b96064.0000000000000074:1b v 127'265, mlcod=119'262)", 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:34.636 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.032660781000001, 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094862829999999999, 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534134+0000", 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534135+0000", 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.607e-06 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534148+0000", 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2136e-05 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534172+0000", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4416000000000001e-05 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534192+0000", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9779000000000002e-05 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534350+0000", 2024-09-11T05:53:34.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157973 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.534410+0000", 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0591000000000003e-05 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.543447+0000", 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090364490000000002 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.543620+0000", 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173332 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3123 3.1d e127/83 3:bd62314a:::rbd_data.112419b96064.000000000000000a:1b v 127'267, mlcod=119'262)", 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.031035401, 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088280100000000007, 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.641 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535759+0000", 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535761+0000", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5030000000000001e-06 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535772+0000", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1212999999999999e-05 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535795+0000", 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3353000000000002e-05 2024-09-11T05:53:34.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.535814+0000", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8243000000000001e-05 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.539424+0000", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036103630000000001 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.539489+0000", 2024-09-11T05:53:34.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4951999999999997e-05 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.544500+0000", 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050110959999999996 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.544587+0000", 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7287000000000006e-05 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:166934 3.10 3:0eb31573:::rbd_header.112419b96064:head [watch unwatch cookie 140146796456272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.011309215000001, 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009557701, 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 166934 2024-09-11T05:53:34.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555485+0000", 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555487+0000", 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555496+0000", 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.524e-06 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555518+0000", 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2393000000000001e-05 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.555535+0000", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7088999999999999e-05 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.556380+0000", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00084499399999999995 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.556622+0000", 2024-09-11T05:53:34.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024143999999999999 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.557013+0000", 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039166 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.561961+0000", 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049480169999999999 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.564779+0000", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028181690000000001 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.564871+0000", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1836999999999995e-05 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:05.565043+0000", 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171872 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.651 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:167482 3.10 3:0ee748d8:::rbd_data.11242d2cfbf4.0000000000000009:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.751455647, 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011179168, 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:34.652 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 167482 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815339+0000", 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815343+0000", 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0400000000000003e-06 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815363+0000", 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9933e-05 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815406+0000", 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3123e-05 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815434+0000", 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7659000000000001e-05 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815579+0000", 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014556700000000001 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.815591+0000", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1609e-05 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.822363+0000", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067724400000000002 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.822802+0000", 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043906199999999999 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.823130+0000", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032797600000000001 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.825983+0000", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028525949999999999 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826209+0000", 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022627500000000001 2024-09-11T05:53:34.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826362+0000", 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015248 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:11.826518+0000", 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015640900000000001 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4388.0:167964 3.d e136/83 3:b6a339dc:::rbd_data.1124fc73d6eb.000000000000000a:head v 136'524, mlcod=135'522)", 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.750222093000001, 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012102942, 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:34.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816573+0000", 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7700000000000003e-07 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.816585+0000", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1873e-05 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.819016+0000", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024302849999999999 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.819063+0000", 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7652999999999999e-05 2024-09-11T05:53:34.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.822644+0000", 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035804180000000001 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.822699+0000", 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4892000000000002e-05 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.828556+0000", 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058579269999999998 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:14.828675+0000", 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119017 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:3293 3.a e138/28 3:55ad635f:::rbd_data.1124c865af2a.0000000000000003:13 v 138'711, mlcod=137'708)", 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:34.663 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.851864337999999, 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010238209999999999, 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714930+0000", 2024-09-11T05:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714931+0000", 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6499999999999999e-07 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714938+0000", 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.481e-06 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:34.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714968+0000", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9456000000000001e-05 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.714981+0000", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3413e-05 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.715073+0000", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1464000000000004e-05 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.715103+0000", 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0536e-05 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.725095+0000", 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099921300000000001 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:53:16.725168+0000", 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3264999999999997e-05 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:34.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:34.668 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-11T05:53:34.797 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.798 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:34.798 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-11T05:53:34.798 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.813 INFO:teuthology.orchestra.run.smithi165.stderr:marked in osd.5. 2024-09-11T05:53:34.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-11T05:53:34.836 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-11T05:53:34.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-11T05:53:34.937 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:34.937 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-11T05:53:34.937 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-11T05:53:34.937 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-11T05:53:34.937 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:34.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-11T05:53:34.961 INFO:teuthology.orchestra.run.smithi165.stdout:mon.a: {} 2024-09-11T05:53:34.961 INFO:teuthology.orchestra.run.smithi165.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:53:34.972 INFO:teuthology.orchestra.run.smithi165.stdout:mon.b: {} 2024-09-11T05:53:34.973 INFO:teuthology.orchestra.run.smithi165.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:53:34.981 INFO:teuthology.orchestra.run.smithi165.stdout:mon.c: {} 2024-09-11T05:53:34.982 INFO:teuthology.orchestra.run.smithi165.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T05:53:34.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152409 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 3936256~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.585501211, 2024-09-11T05:53:35.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10653425699999999, 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152409 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.488142+0000", 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491118+0000", 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029758440000000001 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491129+0000", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1060000000000001e-05 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491151+0000", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2038000000000001e-05 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491175+0000", 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3955e-05 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594441+0000", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103265886 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594553+0000", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112218 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.594676+0000", 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123256 2024-09-11T05:53:35.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152417 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3330048~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.582208971, 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106754605, 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152417 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491434+0000", 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491435+0000", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6700000000000003e-07 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491438+0000", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1429999999999998e-06 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491447+0000", 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5890000000000003e-06 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491457+0000", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0463e-05 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597827+0000", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106369902 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.597989+0000", 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162552 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598189+0000", 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019928900000000001 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152418 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3391488~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.582173052999998, 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107142056, 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152418 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.491470+0000", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495385+0000", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039145459999999997 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495396+0000", 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1412e-05 2024-09-11T05:53:35.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495419+0000", 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2673000000000001e-05 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.495456+0000", 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7363999999999997e-05 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598253+0000", 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102797363 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598416+0000", 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016307600000000001 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.598612+0000", 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019562200000000001 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152428 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3821568~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.577442601999998, 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10670555800000001, 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.089 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152428 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496200+0000", 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496202+0000", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.26e-06 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496208+0000", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.19e-06 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496229+0000", 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1382e-05 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496249+0000", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806e-05 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602575+0000", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106325935 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602717+0000", 2024-09-11T05:53:35.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142142 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602906+0000", 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018884300000000001 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152429 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3358720~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:35.093 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.577367649000003, 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10707137799999999, 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152429 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:35.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496275+0000", 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496277+0000", 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0159999999999998e-06 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496284+0000", 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4069999999999998e-06 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496304+0000", 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0409000000000001e-05 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496320+0000", 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6039e-05 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.602989+0000", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106668918 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603156+0000", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166324 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603347+0000", 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019126500000000001 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152430 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3883008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.577301419999998, 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10735383599999999, 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152430 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.098 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496342+0000", 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496343+0000", 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3239999999999999e-06 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496349+0000", 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7089999999999999e-06 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496369+0000", 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0046000000000001e-05 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496386+0000", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7399000000000001e-05 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603416+0000", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107030103 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603538+0000", 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012149700000000001 2024-09-11T05:53:35.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603695+0000", 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157758 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152431 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3420160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.577234869999998, 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107738669, 2024-09-11T05:53:35.102 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152431 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.496408+0000", 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499158+0000", 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027497960000000001 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499168+0000", 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0287e-05 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499198+0000", 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9331000000000001e-05 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499217+0000", 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8955999999999999e-05 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603776+0000", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104558994 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.603928+0000", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015240300000000001 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.604147+0000", 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021890200000000001 2024-09-11T05:53:35.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152436 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4067328~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.574111633999998, 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106657074, 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152436 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499531+0000", 2024-09-11T05:53:35.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499533+0000", 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.24e-06 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499538+0000", 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7529999999999998e-06 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499556+0000", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7469000000000002e-05 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499571+0000", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5079000000000001e-05 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.605909+0000", 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633767500000001 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606028+0000", 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119623 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606189+0000", 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160235 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152437 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4128768~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.111 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.574020746000002, 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106988261, 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152437 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499622+0000", 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499624+0000", 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.424e-06 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499630+0000", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499657+0000", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7341000000000001e-05 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499673+0000", 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5435999999999999e-05 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606265+0000", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106591884 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606418+0000", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015324100000000001 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606611+0000", 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192652 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152438 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3604480~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.573945723000001, 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107268207, 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152438 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499697+0000", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499699+0000", 2024-09-11T05:53:35.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-06 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499705+0000", 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499723+0000", 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079000000000001e-05 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499746+0000", 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606687+0000", 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10694168799999999 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606808+0000", 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120874 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.606966+0000", 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157288 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152439 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4190208~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.573874316999998, 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10763426600000001, 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152439 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.499769+0000", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503360+0000", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035916199999999998 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503370+0000", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0063e-05 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503393+0000", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2135999999999999e-05 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503411+0000", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8091999999999999e-05 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607040+0000", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10362929999999999 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607201+0000", 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000161471 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.607403+0000", 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020158400000000001 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152446 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3850240~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569743623000001, 2024-09-11T05:53:35.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106538883, 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152446 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:35.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503899+0000", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503901+0000", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1570000000000001e-06 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503906+0000", 2024-09-11T05:53:35.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.519e-06 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503925+0000", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842999999999999e-05 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503942+0000", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6725e-05 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610116+0000", 2024-09-11T05:53:35.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106174781 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610257+0000", 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014074199999999999 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610438+0000", 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018111599999999999 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152447 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 241664~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569680347000002, 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106738712, 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152447 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503963+0000", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503964+0000", 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0139999999999999e-06 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.503980+0000", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6518999999999999e-05 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504000+0000", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0179000000000001e-05 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504016+0000", 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5896000000000002e-05 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610477+0000", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106461047 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610576+0000", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8189000000000005e-05 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610701+0000", 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012586800000000001 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152448 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3911680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569607556999998, 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10704807199999999, 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152448 2024-09-11T05:53:35.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504036+0000", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504037+0000", 2024-09-11T05:53:35.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1459999999999999e-06 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504042+0000", 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4569999999999996e-06 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504063+0000", 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0672000000000001e-05 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504087+0000", 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3926000000000001e-05 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610774+0000", 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106687405 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.610898+0000", 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123666 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611084+0000", 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018579999999999999 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152449 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 303104~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569534750999999, 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107235099, 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152449 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504108+0000", 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504110+0000", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1799999999999999e-06 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504115+0000", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9259999999999997e-06 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504133+0000", 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7405999999999999e-05 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504147+0000", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4346e-05 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611123+0000", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106975898 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611216+0000", 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2774000000000003e-05 2024-09-11T05:53:35.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611343+0000", 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012756900000000001 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152450 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 364544~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569474618000001, 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10755002800000001, 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152450 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504168+0000", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504170+0000", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504175+0000", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960000000000002e-06 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504193+0000", 2024-09-11T05:53:35.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7506000000000001e-05 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504226+0000", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611421+0000", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107195228 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611547+0000", 2024-09-11T05:53:35.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012559 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611718+0000", 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017145999999999999 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152451 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 3973120~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.569383549999998, 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107737862, 2024-09-11T05:53:35.145 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152451 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.504260+0000", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507092+0000", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028320189999999999 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507102+0000", 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0016e-05 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507124+0000", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2132000000000001e-05 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507147+0000", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3649e-05 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611784+0000", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.104637115 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611879+0000", 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4042000000000005e-05 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.611997+0000", 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011888900000000001 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152458 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 671744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.566058898000001, 2024-09-11T05:53:35.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106628267, 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152458 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507584+0000", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507585+0000", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.133e-06 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507591+0000", 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1330000000000004e-06 2024-09-11T05:53:35.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507628+0000", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6807999999999997e-05 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507643+0000", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4816e-05 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.613998+0000", 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10635518300000001 2024-09-11T05:53:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614090+0000", 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2077999999999995e-05 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614212+0000", 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012211599999999999 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.153 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152459 3.13 3:c883ded2:::rbd_data.1124fd5e4ce2.0000000000000003:24 [read 4157440~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.565948398000003, 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.106835874, 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152459 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507695+0000", 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507696+0000", 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2470000000000001e-06 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507713+0000", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7008999999999999e-05 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507731+0000", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8445999999999999e-05 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507749+0000", 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7625000000000002e-05 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614276+0000", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10652693100000001 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614386+0000", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110494 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614531+0000", 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144122 2024-09-11T05:53:35.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4388.0:152460 3.13 3:ca7c8899:::rbd_data.1124fd5e4ce2.0000000000000004:24 [read 733184~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e99)", 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.565870674999999, 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.107024894, 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "continuous": false, 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4388", 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.165:0/2475937137", 2024-09-11T05:53:35.158 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 152460 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.507772+0000", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511366+0000", 2024-09-11T05:53:35.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035938900000000002 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511376+0000", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0169999999999999e-05 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511400+0000", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3705000000000001e-05 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.511418+0000", 2024-09-11T05:53:35.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8172000000000002e-05 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614587+0000", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10316895800000001 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614678+0000", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1107999999999996e-05 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-11T05:52:37.614797+0000", 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118891 2024-09-11T05:53:35.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-11T05:53:35.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-11T05:53:35.162 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-11T05:53:35.530 INFO:teuthology.orchestra.run.smithi165.stderr:noscrub is unset 2024-09-11T05:53:35.546 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T05:53:35.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T05:53:35.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-11T05:53:36.533 INFO:teuthology.orchestra.run.smithi165.stderr:nodeep-scrub is unset 2024-09-11T05:53:36.533 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T05:53:36.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T05:53:37.839 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T05:53:37.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-11T05:53:38.851 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T05:53:38.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T05:53:39.857 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T05:53:39.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-11T05:53:40.532 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-11T05:53:40.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T05:53:41.535 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T05:53:41.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-11T05:53:42.558 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-11T05:53:42.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T05:53:43.548 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T05:53:43.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-11T05:53:44.885 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-11T05:53:44.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T05:53:45.542 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T05:53:45.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-11T05:53:46.543 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-11T05:53:46.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-11T05:53:47.541 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.6 to 1 (10000) 2024-09-11T05:53:47.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-11T05:53:48.552 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-11T05:53:48.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-11T05:53:49.555 INFO:teuthology.orchestra.run.smithi165.stderr:reweighted osd.7 to 1 (10000) 2024-09-11T05:53:49.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-11T05:53:50.579 INFO:teuthology.orchestra.run.smithi165.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-11T05:53:50.601 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-11T05:53:50.601 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-11T05:53:50.602 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-11T05:53:50.602 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-11T05:53:50.602 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T05:53:50.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:53:50.894 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:50.894 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":171,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:53:50.563505+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:53:34.802739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:50:54.170388+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-11T05:51:04.331649+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:53:50.912 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T05:53:50.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:53:51.198 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:51.198 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":171,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:53:50.563505+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:53:34.802739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:50:54.170388+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-11T05:51:04.331649+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:53:51.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T05:53:51.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T05:53:51.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T05:53:51.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T05:53:51.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T05:53:51.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T05:53:51.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-11T05:53:51.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-11T05:53:51.465 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:51.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:53:51.468 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:51.469 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:51.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:53:51.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:53:51.486 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:51.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:53:51.490 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:51.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:53:51.505 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:51.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:53:51.519 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:51.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T05:53:51.523 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:51.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T05:53:51.969 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:51.988 INFO:teuthology.orchestra.run.smithi165.stdout:60129542183 2024-09-11T05:53:51.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.5 2024-09-11T05:53:52.003 INFO:teuthology.orchestra.run.smithi165.stdout:60129542183 2024-09-11T05:53:52.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542183 for osd.1 2024-09-11T05:53:52.016 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:52.023 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:52.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542183 for osd.2 2024-09-11T05:53:52.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542185 for osd.0 2024-09-11T05:53:52.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.3 2024-09-11T05:53:52.053 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:52.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542185 for osd.4 2024-09-11T05:53:52.073 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:52.077 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:52.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542185 for osd.6 2024-09-11T05:53:52.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.7 2024-09-11T05:53:52.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:53:53.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:53:53.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:53:53.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:53:53.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:53:53.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:53:53.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T05:53:53.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T05:53:53.494 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:53.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542185 for osd.5 2024-09-11T05:53:53.518 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.529 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:53.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542184 for osd.2 2024-09-11T05:53:53.547 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.549 INFO:teuthology.orchestra.run.smithi165.stdout:60129542184 2024-09-11T05:53:53.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542184 for osd.1 2024-09-11T05:53:53.574 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.590 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:53.599 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:53.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542185 for osd.3 2024-09-11T05:53:53.617 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542186 for osd.0 2024-09-11T05:53:53.619 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.619 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:53.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542186 for osd.4 2024-09-11T05:53:53.636 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.667 INFO:teuthology.orchestra.run.smithi165.stdout:60129542185 2024-09-11T05:53:53.674 INFO:teuthology.orchestra.run.smithi165.stdout:60129542186 2024-09-11T05:53:53.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542185 for osd.7 2024-09-11T05:53:53.688 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542186 got 60129542186 for osd.6 2024-09-11T05:53:53.694 DEBUG:teuthology.parallel:result is None 2024-09-11T05:53:53.694 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T05:53:53.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:53:54.007 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:54.017 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:53:54.023 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-11T05:53:52.538514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20240,"ondisk_log_size":20240,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17168,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17580032,"data_stored":9009814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.993908"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3764,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:32:49.666836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3803,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578305+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578305+0000","last_peered":"2024-09-11T05:53:50.578305+0000","last_clean":"2024-09-11T05:53:50.578305+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578305+0000","last_undegraded":"2024-09-11T05:53:50.578305+0000","last_fullsized":"2024-09-11T05:53:50.578305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:37.294197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3665,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321222+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321222+0000","last_peered":"2024-09-11T05:53:51.321222+0000","last_clean":"2024-09-11T05:53:51.321222+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321222+0000","last_undegraded":"2024-09-11T05:53:51.321222+0000","last_fullsized":"2024-09-11T05:53:51.321222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:58.908226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4074,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321097+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321097+0000","last_peered":"2024-09-11T05:53:51.321097+0000","last_clean":"2024-09-11T05:53:51.321097+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321097+0000","last_undegraded":"2024-09-11T05:53:51.321097+0000","last_fullsized":"2024-09-11T05:53:51.321097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:58.745818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"150'575","reported_seq":7978,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984290+0000","last_change":"2024-09-11T05:53:32.559312+0000","last_active":"2024-09-11T05:53:50.984290+0000","last_peered":"2024-09-11T05:53:50.984290+0000","last_clean":"2024-09-11T05:53:50.984290+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:50.984290+0000","last_undegraded":"2024-09-11T05:53:50.984290+0000","last_fullsized":"2024-09-11T05:53:50.984290+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:29.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"153'413","reported_seq":8560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578055+0000","last_change":"2024-09-11T05:53:32.577450+0000","last_active":"2024-09-11T05:53:50.578055+0000","last_peered":"2024-09-11T05:53:50.578055+0000","last_clean":"2024-09-11T05:53:50.578055+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:50.578055+0000","last_undegraded":"2024-09-11T05:53:50.578055+0000","last_fullsized":"2024-09-11T05:53:50.578055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:07.172841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4747,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392118+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392118+0000","last_peered":"2024-09-11T05:53:51.392118+0000","last_clean":"2024-09-11T05:53:51.392118+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392118+0000","last_undegraded":"2024-09-11T05:53:51.392118+0000","last_fullsized":"2024-09-11T05:53:51.392118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:08.354943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8078,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577397+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577397+0000","last_peered":"2024-09-11T05:53:50.577397+0000","last_clean":"2024-09-11T05:53:50.577397+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577397+0000","last_undegraded":"2024-09-11T05:53:50.577397+0000","last_fullsized":"2024-09-11T05:53:50.577397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:37.899979+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4559,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984527+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984527+0000","last_peered":"2024-09-11T05:53:50.984527+0000","last_clean":"2024-09-11T05:53:50.984527+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984527+0000","last_undegraded":"2024-09-11T05:53:50.984527+0000","last_fullsized":"2024-09-11T05:53:50.984527+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:24.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321481+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321481+0000","last_peered":"2024-09-11T05:53:51.321481+0000","last_clean":"2024-09-11T05:53:51.321481+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321481+0000","last_undegraded":"2024-09-11T05:53:51.321481+0000","last_fullsized":"2024-09-11T05:53:51.321481+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:38.995333+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578341+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578341+0000","last_peered":"2024-09-11T05:53:50.578341+0000","last_clean":"2024-09-11T05:53:50.578341+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578341+0000","last_undegraded":"2024-09-11T05:53:50.578341+0000","last_fullsized":"2024-09-11T05:53:50.578341+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:08:39.964622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"154'820","reported_seq":6762,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:32.559340+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:13.214059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392242+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:53:51.392242+0000","last_peered":"2024-09-11T05:53:51.392242+0000","last_clean":"2024-09-11T05:53:51.392242+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:51.392242+0000","last_undegraded":"2024-09-11T05:53:51.392242+0000","last_fullsized":"2024-09-11T05:53:51.392242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578559+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578559+0000","last_peered":"2024-09-11T05:53:50.578559+0000","last_clean":"2024-09-11T05:53:50.578559+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578559+0000","last_undegraded":"2024-09-11T05:53:50.578559+0000","last_fullsized":"2024-09-11T05:53:50.578559+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:34.647509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6298,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392180+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392180+0000","last_peered":"2024-09-11T05:53:51.392180+0000","last_clean":"2024-09-11T05:53:51.392180+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392180+0000","last_undegraded":"2024-09-11T05:53:51.392180+0000","last_fullsized":"2024-09-11T05:53:51.392180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:12.588217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392271+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392271+0000","last_peered":"2024-09-11T05:53:51.392271+0000","last_clean":"2024-09-11T05:53:51.392271+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392271+0000","last_undegraded":"2024-09-11T05:53:51.392271+0000","last_fullsized":"2024-09-11T05:53:51.392271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:52.949525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321448+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321448+0000","last_peered":"2024-09-11T05:53:51.321448+0000","last_clean":"2024-09-11T05:53:51.321448+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321448+0000","last_undegraded":"2024-09-11T05:53:51.321448+0000","last_fullsized":"2024-09-11T05:53:51.321448+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4626,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578179+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578179+0000","last_peered":"2024-09-11T05:53:50.578179+0000","last_clean":"2024-09-11T05:53:50.578179+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578179+0000","last_undegraded":"2024-09-11T05:53:50.578179+0000","last_fullsized":"2024-09-11T05:53:50.578179+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:37.696649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.983124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"152'671","reported_seq":3519,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392082+0000","last_change":"2024-09-11T05:53:32.544410+0000","last_active":"2024-09-11T05:53:51.392082+0000","last_peered":"2024-09-11T05:53:51.392082+0000","last_clean":"2024-09-11T05:53:51.392082+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:51.392082+0000","last_undegraded":"2024-09-11T05:53:51.392082+0000","last_fullsized":"2024-09-11T05:53:51.392082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.086523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":317,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577197+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:53:50.577197+0000","last_peered":"2024-09-11T05:53:50.577197+0000","last_clean":"2024-09-11T05:53:50.577197+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:50.577197+0000","last_undegraded":"2024-09-11T05:53:50.577197+0000","last_fullsized":"2024-09-11T05:53:50.577197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984654+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984654+0000","last_peered":"2024-09-11T05:53:50.984654+0000","last_clean":"2024-09-11T05:53:50.984654+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984654+0000","last_undegraded":"2024-09-11T05:53:50.984654+0000","last_fullsized":"2024-09-11T05:53:50.984654+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:19:38.858978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7404,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577512+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577512+0000","last_peered":"2024-09-11T05:53:50.577512+0000","last_clean":"2024-09-11T05:53:50.577512+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577512+0000","last_undegraded":"2024-09-11T05:53:50.577512+0000","last_fullsized":"2024-09-11T05:53:50.577512+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:27.660765+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.555476+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:33.979145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577619+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577619+0000","last_peered":"2024-09-11T05:53:50.577619+0000","last_clean":"2024-09-11T05:53:50.577619+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577619+0000","last_undegraded":"2024-09-11T05:53:50.577619+0000","last_fullsized":"2024-09-11T05:53:50.577619+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:38:32.189314+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3677,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578121+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578121+0000","last_peered":"2024-09-11T05:53:50.578121+0000","last_clean":"2024-09-11T05:53:50.578121+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578121+0000","last_undegraded":"2024-09-11T05:53:50.578121+0000","last_fullsized":"2024-09-11T05:53:50.578121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:57.790323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8007,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577878+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577878+0000","last_peered":"2024-09-11T05:53:50.577878+0000","last_clean":"2024-09-11T05:53:50.577878+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577878+0000","last_undegraded":"2024-09-11T05:53:50.577878+0000","last_fullsized":"2024-09-11T05:53:50.577878+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:03:44.318939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9449,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321147+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321147+0000","last_peered":"2024-09-11T05:53:51.321147+0000","last_clean":"2024-09-11T05:53:51.321147+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321147+0000","last_undegraded":"2024-09-11T05:53:51.321147+0000","last_fullsized":"2024-09-11T05:53:51.321147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:58.857313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4038,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984586+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984586+0000","last_peered":"2024-09-11T05:53:50.984586+0000","last_clean":"2024-09-11T05:53:50.984586+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984586+0000","last_undegraded":"2024-09-11T05:53:50.984586+0000","last_fullsized":"2024-09-11T05:53:50.984586+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:20:03.379238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20120,"ondisk_log_size":20120,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69284,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302556,"statfs":{"total":96636764160,"available":96565817344,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2043904,"data_stored":977172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41524,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330316,"statfs":{"total":96636764160,"available":96594243584,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2052096,"data_stored":977210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:53:54.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:53:54.309 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:54.310 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:53:54.326 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-11T05:53:52.538514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20240,"ondisk_log_size":20240,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17168,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17580032,"data_stored":9009814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.993908"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3764,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:32:49.666836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3803,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578305+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578305+0000","last_peered":"2024-09-11T05:53:50.578305+0000","last_clean":"2024-09-11T05:53:50.578305+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578305+0000","last_undegraded":"2024-09-11T05:53:50.578305+0000","last_fullsized":"2024-09-11T05:53:50.578305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:37.294197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3665,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321222+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321222+0000","last_peered":"2024-09-11T05:53:51.321222+0000","last_clean":"2024-09-11T05:53:51.321222+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321222+0000","last_undegraded":"2024-09-11T05:53:51.321222+0000","last_fullsized":"2024-09-11T05:53:51.321222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:58.908226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4074,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321097+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321097+0000","last_peered":"2024-09-11T05:53:51.321097+0000","last_clean":"2024-09-11T05:53:51.321097+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321097+0000","last_undegraded":"2024-09-11T05:53:51.321097+0000","last_fullsized":"2024-09-11T05:53:51.321097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:58.745818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"150'575","reported_seq":7978,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984290+0000","last_change":"2024-09-11T05:53:32.559312+0000","last_active":"2024-09-11T05:53:50.984290+0000","last_peered":"2024-09-11T05:53:50.984290+0000","last_clean":"2024-09-11T05:53:50.984290+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:50.984290+0000","last_undegraded":"2024-09-11T05:53:50.984290+0000","last_fullsized":"2024-09-11T05:53:50.984290+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:29.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"153'413","reported_seq":8560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578055+0000","last_change":"2024-09-11T05:53:32.577450+0000","last_active":"2024-09-11T05:53:50.578055+0000","last_peered":"2024-09-11T05:53:50.578055+0000","last_clean":"2024-09-11T05:53:50.578055+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:50.578055+0000","last_undegraded":"2024-09-11T05:53:50.578055+0000","last_fullsized":"2024-09-11T05:53:50.578055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:07.172841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4747,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392118+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392118+0000","last_peered":"2024-09-11T05:53:51.392118+0000","last_clean":"2024-09-11T05:53:51.392118+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392118+0000","last_undegraded":"2024-09-11T05:53:51.392118+0000","last_fullsized":"2024-09-11T05:53:51.392118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:08.354943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8078,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577397+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577397+0000","last_peered":"2024-09-11T05:53:50.577397+0000","last_clean":"2024-09-11T05:53:50.577397+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577397+0000","last_undegraded":"2024-09-11T05:53:50.577397+0000","last_fullsized":"2024-09-11T05:53:50.577397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:37.899979+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4559,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984527+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984527+0000","last_peered":"2024-09-11T05:53:50.984527+0000","last_clean":"2024-09-11T05:53:50.984527+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984527+0000","last_undegraded":"2024-09-11T05:53:50.984527+0000","last_fullsized":"2024-09-11T05:53:50.984527+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:24.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321481+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321481+0000","last_peered":"2024-09-11T05:53:51.321481+0000","last_clean":"2024-09-11T05:53:51.321481+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321481+0000","last_undegraded":"2024-09-11T05:53:51.321481+0000","last_fullsized":"2024-09-11T05:53:51.321481+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:38.995333+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578341+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578341+0000","last_peered":"2024-09-11T05:53:50.578341+0000","last_clean":"2024-09-11T05:53:50.578341+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578341+0000","last_undegraded":"2024-09-11T05:53:50.578341+0000","last_fullsized":"2024-09-11T05:53:50.578341+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:08:39.964622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"154'820","reported_seq":6762,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:32.559340+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:13.214059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392242+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:53:51.392242+0000","last_peered":"2024-09-11T05:53:51.392242+0000","last_clean":"2024-09-11T05:53:51.392242+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:51.392242+0000","last_undegraded":"2024-09-11T05:53:51.392242+0000","last_fullsized":"2024-09-11T05:53:51.392242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578559+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578559+0000","last_peered":"2024-09-11T05:53:50.578559+0000","last_clean":"2024-09-11T05:53:50.578559+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578559+0000","last_undegraded":"2024-09-11T05:53:50.578559+0000","last_fullsized":"2024-09-11T05:53:50.578559+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:34.647509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6298,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392180+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392180+0000","last_peered":"2024-09-11T05:53:51.392180+0000","last_clean":"2024-09-11T05:53:51.392180+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392180+0000","last_undegraded":"2024-09-11T05:53:51.392180+0000","last_fullsized":"2024-09-11T05:53:51.392180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:12.588217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392271+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392271+0000","last_peered":"2024-09-11T05:53:51.392271+0000","last_clean":"2024-09-11T05:53:51.392271+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392271+0000","last_undegraded":"2024-09-11T05:53:51.392271+0000","last_fullsized":"2024-09-11T05:53:51.392271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:52.949525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321448+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321448+0000","last_peered":"2024-09-11T05:53:51.321448+0000","last_clean":"2024-09-11T05:53:51.321448+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321448+0000","last_undegraded":"2024-09-11T05:53:51.321448+0000","last_fullsized":"2024-09-11T05:53:51.321448+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4626,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578179+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578179+0000","last_peered":"2024-09-11T05:53:50.578179+0000","last_clean":"2024-09-11T05:53:50.578179+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578179+0000","last_undegraded":"2024-09-11T05:53:50.578179+0000","last_fullsized":"2024-09-11T05:53:50.578179+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:37.696649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.983124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"152'671","reported_seq":3519,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392082+0000","last_change":"2024-09-11T05:53:32.544410+0000","last_active":"2024-09-11T05:53:51.392082+0000","last_peered":"2024-09-11T05:53:51.392082+0000","last_clean":"2024-09-11T05:53:51.392082+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:51.392082+0000","last_undegraded":"2024-09-11T05:53:51.392082+0000","last_fullsized":"2024-09-11T05:53:51.392082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.086523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":317,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577197+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:53:50.577197+0000","last_peered":"2024-09-11T05:53:50.577197+0000","last_clean":"2024-09-11T05:53:50.577197+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:50.577197+0000","last_undegraded":"2024-09-11T05:53:50.577197+0000","last_fullsized":"2024-09-11T05:53:50.577197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984654+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984654+0000","last_peered":"2024-09-11T05:53:50.984654+0000","last_clean":"2024-09-11T05:53:50.984654+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984654+0000","last_undegraded":"2024-09-11T05:53:50.984654+0000","last_fullsized":"2024-09-11T05:53:50.984654+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:19:38.858978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7404,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577512+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577512+0000","last_peered":"2024-09-11T05:53:50.577512+0000","last_clean":"2024-09-11T05:53:50.577512+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577512+0000","last_undegraded":"2024-09-11T05:53:50.577512+0000","last_fullsized":"2024-09-11T05:53:50.577512+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:27.660765+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.555476+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:33.979145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577619+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577619+0000","last_peered":"2024-09-11T05:53:50.577619+0000","last_clean":"2024-09-11T05:53:50.577619+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577619+0000","last_undegraded":"2024-09-11T05:53:50.577619+0000","last_fullsized":"2024-09-11T05:53:50.577619+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:38:32.189314+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3677,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578121+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578121+0000","last_peered":"2024-09-11T05:53:50.578121+0000","last_clean":"2024-09-11T05:53:50.578121+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578121+0000","last_undegraded":"2024-09-11T05:53:50.578121+0000","last_fullsized":"2024-09-11T05:53:50.578121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:57.790323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8007,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577878+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577878+0000","last_peered":"2024-09-11T05:53:50.577878+0000","last_clean":"2024-09-11T05:53:50.577878+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577878+0000","last_undegraded":"2024-09-11T05:53:50.577878+0000","last_fullsized":"2024-09-11T05:53:50.577878+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:03:44.318939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9449,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321147+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321147+0000","last_peered":"2024-09-11T05:53:51.321147+0000","last_clean":"2024-09-11T05:53:51.321147+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321147+0000","last_undegraded":"2024-09-11T05:53:51.321147+0000","last_fullsized":"2024-09-11T05:53:51.321147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:58.857313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4038,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984586+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984586+0000","last_peered":"2024-09-11T05:53:50.984586+0000","last_clean":"2024-09-11T05:53:50.984586+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984586+0000","last_undegraded":"2024-09-11T05:53:50.984586+0000","last_fullsized":"2024-09-11T05:53:50.984586+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:20:03.379238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20120,"ondisk_log_size":20120,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69284,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302556,"statfs":{"total":96636764160,"available":96565817344,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2043904,"data_stored":977172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41524,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330316,"statfs":{"total":96636764160,"available":96594243584,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2052096,"data_stored":977210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:53:54.328 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T05:53:54.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-11T05:53:54.631 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T05:53:54.640 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:53:54.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:53:54.924 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:54.925 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:53:54.941 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-11T05:53:54.539160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20240,"ondisk_log_size":20240,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17180,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17592320,"data_stored":9015627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001842"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3764,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:32:49.666836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3803,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578305+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578305+0000","last_peered":"2024-09-11T05:53:50.578305+0000","last_clean":"2024-09-11T05:53:50.578305+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578305+0000","last_undegraded":"2024-09-11T05:53:50.578305+0000","last_fullsized":"2024-09-11T05:53:50.578305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:37.294197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3665,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321222+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321222+0000","last_peered":"2024-09-11T05:53:51.321222+0000","last_clean":"2024-09-11T05:53:51.321222+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321222+0000","last_undegraded":"2024-09-11T05:53:51.321222+0000","last_fullsized":"2024-09-11T05:53:51.321222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:58.908226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4074,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321097+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321097+0000","last_peered":"2024-09-11T05:53:51.321097+0000","last_clean":"2024-09-11T05:53:51.321097+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321097+0000","last_undegraded":"2024-09-11T05:53:51.321097+0000","last_fullsized":"2024-09-11T05:53:51.321097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:58.745818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"150'575","reported_seq":7978,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984290+0000","last_change":"2024-09-11T05:53:32.559312+0000","last_active":"2024-09-11T05:53:50.984290+0000","last_peered":"2024-09-11T05:53:50.984290+0000","last_clean":"2024-09-11T05:53:50.984290+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:50.984290+0000","last_undegraded":"2024-09-11T05:53:50.984290+0000","last_fullsized":"2024-09-11T05:53:50.984290+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:29.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"153'413","reported_seq":8560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578055+0000","last_change":"2024-09-11T05:53:32.577450+0000","last_active":"2024-09-11T05:53:50.578055+0000","last_peered":"2024-09-11T05:53:50.578055+0000","last_clean":"2024-09-11T05:53:50.578055+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:50.578055+0000","last_undegraded":"2024-09-11T05:53:50.578055+0000","last_fullsized":"2024-09-11T05:53:50.578055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:07.172841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4747,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392118+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392118+0000","last_peered":"2024-09-11T05:53:51.392118+0000","last_clean":"2024-09-11T05:53:51.392118+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392118+0000","last_undegraded":"2024-09-11T05:53:51.392118+0000","last_fullsized":"2024-09-11T05:53:51.392118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:08.354943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8078,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577397+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577397+0000","last_peered":"2024-09-11T05:53:50.577397+0000","last_clean":"2024-09-11T05:53:50.577397+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577397+0000","last_undegraded":"2024-09-11T05:53:50.577397+0000","last_fullsized":"2024-09-11T05:53:50.577397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:37.899979+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4559,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984527+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984527+0000","last_peered":"2024-09-11T05:53:50.984527+0000","last_clean":"2024-09-11T05:53:50.984527+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984527+0000","last_undegraded":"2024-09-11T05:53:50.984527+0000","last_fullsized":"2024-09-11T05:53:50.984527+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:24.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321481+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321481+0000","last_peered":"2024-09-11T05:53:51.321481+0000","last_clean":"2024-09-11T05:53:51.321481+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321481+0000","last_undegraded":"2024-09-11T05:53:51.321481+0000","last_fullsized":"2024-09-11T05:53:51.321481+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:38.995333+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578341+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578341+0000","last_peered":"2024-09-11T05:53:50.578341+0000","last_clean":"2024-09-11T05:53:50.578341+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578341+0000","last_undegraded":"2024-09-11T05:53:50.578341+0000","last_fullsized":"2024-09-11T05:53:50.578341+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:08:39.964622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"154'820","reported_seq":6762,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:32.559340+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:13.214059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392242+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:53:51.392242+0000","last_peered":"2024-09-11T05:53:51.392242+0000","last_clean":"2024-09-11T05:53:51.392242+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:51.392242+0000","last_undegraded":"2024-09-11T05:53:51.392242+0000","last_fullsized":"2024-09-11T05:53:51.392242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578559+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578559+0000","last_peered":"2024-09-11T05:53:50.578559+0000","last_clean":"2024-09-11T05:53:50.578559+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578559+0000","last_undegraded":"2024-09-11T05:53:50.578559+0000","last_fullsized":"2024-09-11T05:53:50.578559+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:34.647509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6298,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392180+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392180+0000","last_peered":"2024-09-11T05:53:51.392180+0000","last_clean":"2024-09-11T05:53:51.392180+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392180+0000","last_undegraded":"2024-09-11T05:53:51.392180+0000","last_fullsized":"2024-09-11T05:53:51.392180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:12.588217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392271+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392271+0000","last_peered":"2024-09-11T05:53:51.392271+0000","last_clean":"2024-09-11T05:53:51.392271+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392271+0000","last_undegraded":"2024-09-11T05:53:51.392271+0000","last_fullsized":"2024-09-11T05:53:51.392271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:52.949525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321448+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321448+0000","last_peered":"2024-09-11T05:53:51.321448+0000","last_clean":"2024-09-11T05:53:51.321448+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321448+0000","last_undegraded":"2024-09-11T05:53:51.321448+0000","last_fullsized":"2024-09-11T05:53:51.321448+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4626,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578179+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578179+0000","last_peered":"2024-09-11T05:53:50.578179+0000","last_clean":"2024-09-11T05:53:50.578179+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578179+0000","last_undegraded":"2024-09-11T05:53:50.578179+0000","last_fullsized":"2024-09-11T05:53:50.578179+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:37.696649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.983124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"152'671","reported_seq":3519,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392082+0000","last_change":"2024-09-11T05:53:32.544410+0000","last_active":"2024-09-11T05:53:51.392082+0000","last_peered":"2024-09-11T05:53:51.392082+0000","last_clean":"2024-09-11T05:53:51.392082+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:51.392082+0000","last_undegraded":"2024-09-11T05:53:51.392082+0000","last_fullsized":"2024-09-11T05:53:51.392082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.086523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":317,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577197+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:53:50.577197+0000","last_peered":"2024-09-11T05:53:50.577197+0000","last_clean":"2024-09-11T05:53:50.577197+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:50.577197+0000","last_undegraded":"2024-09-11T05:53:50.577197+0000","last_fullsized":"2024-09-11T05:53:50.577197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984654+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984654+0000","last_peered":"2024-09-11T05:53:50.984654+0000","last_clean":"2024-09-11T05:53:50.984654+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984654+0000","last_undegraded":"2024-09-11T05:53:50.984654+0000","last_fullsized":"2024-09-11T05:53:50.984654+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:19:38.858978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7404,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577512+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577512+0000","last_peered":"2024-09-11T05:53:50.577512+0000","last_clean":"2024-09-11T05:53:50.577512+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577512+0000","last_undegraded":"2024-09-11T05:53:50.577512+0000","last_fullsized":"2024-09-11T05:53:50.577512+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:27.660765+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.555476+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:33.979145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577619+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577619+0000","last_peered":"2024-09-11T05:53:50.577619+0000","last_clean":"2024-09-11T05:53:50.577619+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577619+0000","last_undegraded":"2024-09-11T05:53:50.577619+0000","last_fullsized":"2024-09-11T05:53:50.577619+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:38:32.189314+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3677,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578121+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578121+0000","last_peered":"2024-09-11T05:53:50.578121+0000","last_clean":"2024-09-11T05:53:50.578121+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578121+0000","last_undegraded":"2024-09-11T05:53:50.578121+0000","last_fullsized":"2024-09-11T05:53:50.578121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:57.790323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8007,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577878+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577878+0000","last_peered":"2024-09-11T05:53:50.577878+0000","last_clean":"2024-09-11T05:53:50.577878+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577878+0000","last_undegraded":"2024-09-11T05:53:50.577878+0000","last_fullsized":"2024-09-11T05:53:50.577878+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:03:44.318939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9449,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321147+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321147+0000","last_peered":"2024-09-11T05:53:51.321147+0000","last_clean":"2024-09-11T05:53:51.321147+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321147+0000","last_undegraded":"2024-09-11T05:53:51.321147+0000","last_fullsized":"2024-09-11T05:53:51.321147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:58.857313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4038,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984586+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984586+0000","last_peered":"2024-09-11T05:53:50.984586+0000","last_clean":"2024-09-11T05:53:50.984586+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984586+0000","last_undegraded":"2024-09-11T05:53:50.984586+0000","last_fullsized":"2024-09-11T05:53:50.984586+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:20:03.379238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20120,"ondisk_log_size":20120,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69284,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302556,"statfs":{"total":96636764160,"available":96565817344,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41524,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330316,"statfs":{"total":96636764160,"available":96594243584,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2052096,"data_stored":977210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:53:54.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:53:55.224 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:55.225 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:53:55.241 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-11T05:53:54.539160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20240,"ondisk_log_size":20240,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17180,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17592320,"data_stored":9015627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001842"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3764,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:32:49.666836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3803,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578305+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578305+0000","last_peered":"2024-09-11T05:53:50.578305+0000","last_clean":"2024-09-11T05:53:50.578305+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578305+0000","last_undegraded":"2024-09-11T05:53:50.578305+0000","last_fullsized":"2024-09-11T05:53:50.578305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:37.294197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3665,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321222+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321222+0000","last_peered":"2024-09-11T05:53:51.321222+0000","last_clean":"2024-09-11T05:53:51.321222+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321222+0000","last_undegraded":"2024-09-11T05:53:51.321222+0000","last_fullsized":"2024-09-11T05:53:51.321222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:58.908226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4074,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321097+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321097+0000","last_peered":"2024-09-11T05:53:51.321097+0000","last_clean":"2024-09-11T05:53:51.321097+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321097+0000","last_undegraded":"2024-09-11T05:53:51.321097+0000","last_fullsized":"2024-09-11T05:53:51.321097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:58.745818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"150'575","reported_seq":7978,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984290+0000","last_change":"2024-09-11T05:53:32.559312+0000","last_active":"2024-09-11T05:53:50.984290+0000","last_peered":"2024-09-11T05:53:50.984290+0000","last_clean":"2024-09-11T05:53:50.984290+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:50.984290+0000","last_undegraded":"2024-09-11T05:53:50.984290+0000","last_fullsized":"2024-09-11T05:53:50.984290+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:29.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"153'413","reported_seq":8560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578055+0000","last_change":"2024-09-11T05:53:32.577450+0000","last_active":"2024-09-11T05:53:50.578055+0000","last_peered":"2024-09-11T05:53:50.578055+0000","last_clean":"2024-09-11T05:53:50.578055+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:50.578055+0000","last_undegraded":"2024-09-11T05:53:50.578055+0000","last_fullsized":"2024-09-11T05:53:50.578055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:07.172841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4747,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392118+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392118+0000","last_peered":"2024-09-11T05:53:51.392118+0000","last_clean":"2024-09-11T05:53:51.392118+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392118+0000","last_undegraded":"2024-09-11T05:53:51.392118+0000","last_fullsized":"2024-09-11T05:53:51.392118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:08.354943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8078,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577397+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577397+0000","last_peered":"2024-09-11T05:53:50.577397+0000","last_clean":"2024-09-11T05:53:50.577397+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577397+0000","last_undegraded":"2024-09-11T05:53:50.577397+0000","last_fullsized":"2024-09-11T05:53:50.577397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:37.899979+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4559,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984527+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984527+0000","last_peered":"2024-09-11T05:53:50.984527+0000","last_clean":"2024-09-11T05:53:50.984527+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984527+0000","last_undegraded":"2024-09-11T05:53:50.984527+0000","last_fullsized":"2024-09-11T05:53:50.984527+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:24.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321481+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321481+0000","last_peered":"2024-09-11T05:53:51.321481+0000","last_clean":"2024-09-11T05:53:51.321481+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321481+0000","last_undegraded":"2024-09-11T05:53:51.321481+0000","last_fullsized":"2024-09-11T05:53:51.321481+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:38.995333+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578341+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578341+0000","last_peered":"2024-09-11T05:53:50.578341+0000","last_clean":"2024-09-11T05:53:50.578341+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578341+0000","last_undegraded":"2024-09-11T05:53:50.578341+0000","last_fullsized":"2024-09-11T05:53:50.578341+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:08:39.964622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"154'820","reported_seq":6762,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:32.559340+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:13.214059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392242+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:53:51.392242+0000","last_peered":"2024-09-11T05:53:51.392242+0000","last_clean":"2024-09-11T05:53:51.392242+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:51.392242+0000","last_undegraded":"2024-09-11T05:53:51.392242+0000","last_fullsized":"2024-09-11T05:53:51.392242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578559+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578559+0000","last_peered":"2024-09-11T05:53:50.578559+0000","last_clean":"2024-09-11T05:53:50.578559+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578559+0000","last_undegraded":"2024-09-11T05:53:50.578559+0000","last_fullsized":"2024-09-11T05:53:50.578559+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:34.647509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6298,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392180+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392180+0000","last_peered":"2024-09-11T05:53:51.392180+0000","last_clean":"2024-09-11T05:53:51.392180+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392180+0000","last_undegraded":"2024-09-11T05:53:51.392180+0000","last_fullsized":"2024-09-11T05:53:51.392180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:12.588217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392271+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392271+0000","last_peered":"2024-09-11T05:53:51.392271+0000","last_clean":"2024-09-11T05:53:51.392271+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392271+0000","last_undegraded":"2024-09-11T05:53:51.392271+0000","last_fullsized":"2024-09-11T05:53:51.392271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:52.949525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321448+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321448+0000","last_peered":"2024-09-11T05:53:51.321448+0000","last_clean":"2024-09-11T05:53:51.321448+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321448+0000","last_undegraded":"2024-09-11T05:53:51.321448+0000","last_fullsized":"2024-09-11T05:53:51.321448+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4626,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578179+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578179+0000","last_peered":"2024-09-11T05:53:50.578179+0000","last_clean":"2024-09-11T05:53:50.578179+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578179+0000","last_undegraded":"2024-09-11T05:53:50.578179+0000","last_fullsized":"2024-09-11T05:53:50.578179+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:37.696649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.983124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"152'671","reported_seq":3519,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392082+0000","last_change":"2024-09-11T05:53:32.544410+0000","last_active":"2024-09-11T05:53:51.392082+0000","last_peered":"2024-09-11T05:53:51.392082+0000","last_clean":"2024-09-11T05:53:51.392082+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:51.392082+0000","last_undegraded":"2024-09-11T05:53:51.392082+0000","last_fullsized":"2024-09-11T05:53:51.392082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.086523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":317,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577197+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:53:50.577197+0000","last_peered":"2024-09-11T05:53:50.577197+0000","last_clean":"2024-09-11T05:53:50.577197+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:50.577197+0000","last_undegraded":"2024-09-11T05:53:50.577197+0000","last_fullsized":"2024-09-11T05:53:50.577197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984654+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984654+0000","last_peered":"2024-09-11T05:53:50.984654+0000","last_clean":"2024-09-11T05:53:50.984654+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984654+0000","last_undegraded":"2024-09-11T05:53:50.984654+0000","last_fullsized":"2024-09-11T05:53:50.984654+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:19:38.858978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7404,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577512+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577512+0000","last_peered":"2024-09-11T05:53:50.577512+0000","last_clean":"2024-09-11T05:53:50.577512+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577512+0000","last_undegraded":"2024-09-11T05:53:50.577512+0000","last_fullsized":"2024-09-11T05:53:50.577512+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:27.660765+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.555476+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:33.979145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577619+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577619+0000","last_peered":"2024-09-11T05:53:50.577619+0000","last_clean":"2024-09-11T05:53:50.577619+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577619+0000","last_undegraded":"2024-09-11T05:53:50.577619+0000","last_fullsized":"2024-09-11T05:53:50.577619+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:38:32.189314+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3677,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578121+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578121+0000","last_peered":"2024-09-11T05:53:50.578121+0000","last_clean":"2024-09-11T05:53:50.578121+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578121+0000","last_undegraded":"2024-09-11T05:53:50.578121+0000","last_fullsized":"2024-09-11T05:53:50.578121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:57.790323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8007,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577878+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577878+0000","last_peered":"2024-09-11T05:53:50.577878+0000","last_clean":"2024-09-11T05:53:50.577878+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577878+0000","last_undegraded":"2024-09-11T05:53:50.577878+0000","last_fullsized":"2024-09-11T05:53:50.577878+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:03:44.318939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9449,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321147+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321147+0000","last_peered":"2024-09-11T05:53:51.321147+0000","last_clean":"2024-09-11T05:53:51.321147+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321147+0000","last_undegraded":"2024-09-11T05:53:51.321147+0000","last_fullsized":"2024-09-11T05:53:51.321147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:58.857313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4038,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984586+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984586+0000","last_peered":"2024-09-11T05:53:50.984586+0000","last_clean":"2024-09-11T05:53:50.984586+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984586+0000","last_undegraded":"2024-09-11T05:53:50.984586+0000","last_fullsized":"2024-09-11T05:53:50.984586+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:20:03.379238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20120,"ondisk_log_size":20120,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69284,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302556,"statfs":{"total":96636764160,"available":96565817344,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41524,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330316,"statfs":{"total":96636764160,"available":96594243584,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2052096,"data_stored":977210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:53:55.243 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:53:55.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:53:55.533 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:55.534 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:53:55.550 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-11T05:53:54.539160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20240,"ondisk_log_size":20240,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17180,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17592320,"data_stored":9015627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001842"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3764,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:32:49.666836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3803,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578305+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578305+0000","last_peered":"2024-09-11T05:53:50.578305+0000","last_clean":"2024-09-11T05:53:50.578305+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578305+0000","last_undegraded":"2024-09-11T05:53:50.578305+0000","last_fullsized":"2024-09-11T05:53:50.578305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:11:37.294197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3665,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321222+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321222+0000","last_peered":"2024-09-11T05:53:51.321222+0000","last_clean":"2024-09-11T05:53:51.321222+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321222+0000","last_undegraded":"2024-09-11T05:53:51.321222+0000","last_fullsized":"2024-09-11T05:53:51.321222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:58.908226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4074,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321097+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321097+0000","last_peered":"2024-09-11T05:53:51.321097+0000","last_clean":"2024-09-11T05:53:51.321097+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321097+0000","last_undegraded":"2024-09-11T05:53:51.321097+0000","last_fullsized":"2024-09-11T05:53:51.321097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:58.745818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"150'575","reported_seq":7978,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984290+0000","last_change":"2024-09-11T05:53:32.559312+0000","last_active":"2024-09-11T05:53:50.984290+0000","last_peered":"2024-09-11T05:53:50.984290+0000","last_clean":"2024-09-11T05:53:50.984290+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:50.984290+0000","last_undegraded":"2024-09-11T05:53:50.984290+0000","last_fullsized":"2024-09-11T05:53:50.984290+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:29.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"153'413","reported_seq":8560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578055+0000","last_change":"2024-09-11T05:53:32.577450+0000","last_active":"2024-09-11T05:53:50.578055+0000","last_peered":"2024-09-11T05:53:50.578055+0000","last_clean":"2024-09-11T05:53:50.578055+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:50.578055+0000","last_undegraded":"2024-09-11T05:53:50.578055+0000","last_fullsized":"2024-09-11T05:53:50.578055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:07.172841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4747,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392118+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392118+0000","last_peered":"2024-09-11T05:53:51.392118+0000","last_clean":"2024-09-11T05:53:51.392118+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392118+0000","last_undegraded":"2024-09-11T05:53:51.392118+0000","last_fullsized":"2024-09-11T05:53:51.392118+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:08.354943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8078,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577397+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577397+0000","last_peered":"2024-09-11T05:53:50.577397+0000","last_clean":"2024-09-11T05:53:50.577397+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577397+0000","last_undegraded":"2024-09-11T05:53:50.577397+0000","last_fullsized":"2024-09-11T05:53:50.577397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:37.899979+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4559,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984527+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984527+0000","last_peered":"2024-09-11T05:53:50.984527+0000","last_clean":"2024-09-11T05:53:50.984527+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984527+0000","last_undegraded":"2024-09-11T05:53:50.984527+0000","last_fullsized":"2024-09-11T05:53:50.984527+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:24.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321481+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321481+0000","last_peered":"2024-09-11T05:53:51.321481+0000","last_clean":"2024-09-11T05:53:51.321481+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321481+0000","last_undegraded":"2024-09-11T05:53:51.321481+0000","last_fullsized":"2024-09-11T05:53:51.321481+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:38.995333+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578341+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578341+0000","last_peered":"2024-09-11T05:53:50.578341+0000","last_clean":"2024-09-11T05:53:50.578341+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578341+0000","last_undegraded":"2024-09-11T05:53:50.578341+0000","last_fullsized":"2024-09-11T05:53:50.578341+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:08:39.964622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"154'820","reported_seq":6762,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984528+0000","last_change":"2024-09-11T05:53:32.559340+0000","last_active":"2024-09-11T05:53:50.984528+0000","last_peered":"2024-09-11T05:53:50.984528+0000","last_clean":"2024-09-11T05:53:50.984528+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:50.984528+0000","last_undegraded":"2024-09-11T05:53:50.984528+0000","last_fullsized":"2024-09-11T05:53:50.984528+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:17:13.214059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392242+0000","last_change":"2024-09-11T05:52:05.513640+0000","last_active":"2024-09-11T05:53:51.392242+0000","last_peered":"2024-09-11T05:53:51.392242+0000","last_clean":"2024-09-11T05:53:51.392242+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:51.392242+0000","last_undegraded":"2024-09-11T05:53:51.392242+0000","last_fullsized":"2024-09-11T05:53:51.392242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:02.011324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578559+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578559+0000","last_peered":"2024-09-11T05:53:50.578559+0000","last_clean":"2024-09-11T05:53:50.578559+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578559+0000","last_undegraded":"2024-09-11T05:53:50.578559+0000","last_fullsized":"2024-09-11T05:53:50.578559+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:34.647509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6298,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392180+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392180+0000","last_peered":"2024-09-11T05:53:51.392180+0000","last_clean":"2024-09-11T05:53:51.392180+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392180+0000","last_undegraded":"2024-09-11T05:53:51.392180+0000","last_fullsized":"2024-09-11T05:53:51.392180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:12.588217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392271+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392271+0000","last_peered":"2024-09-11T05:53:51.392271+0000","last_clean":"2024-09-11T05:53:51.392271+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392271+0000","last_undegraded":"2024-09-11T05:53:51.392271+0000","last_fullsized":"2024-09-11T05:53:51.392271+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:52.949525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6190,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321448+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321448+0000","last_peered":"2024-09-11T05:53:51.321448+0000","last_clean":"2024-09-11T05:53:51.321448+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321448+0000","last_undegraded":"2024-09-11T05:53:51.321448+0000","last_fullsized":"2024-09-11T05:53:51.321448+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:54.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4626,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578179+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578179+0000","last_peered":"2024-09-11T05:53:50.578179+0000","last_clean":"2024-09-11T05:53:50.578179+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578179+0000","last_undegraded":"2024-09-11T05:53:50.578179+0000","last_fullsized":"2024-09-11T05:53:50.578179+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:37.696649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:29.983124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"152'671","reported_seq":3519,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392082+0000","last_change":"2024-09-11T05:53:32.544410+0000","last_active":"2024-09-11T05:53:51.392082+0000","last_peered":"2024-09-11T05:53:51.392082+0000","last_clean":"2024-09-11T05:53:51.392082+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:51.392082+0000","last_undegraded":"2024-09-11T05:53:51.392082+0000","last_fullsized":"2024-09-11T05:53:51.392082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:43:11.086523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":317,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577197+0000","last_change":"2024-09-11T05:50:58.167606+0000","last_active":"2024-09-11T05:53:50.577197+0000","last_peered":"2024-09-11T05:53:50.577197+0000","last_clean":"2024-09-11T05:53:50.577197+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:50.577197+0000","last_undegraded":"2024-09-11T05:53:50.577197+0000","last_fullsized":"2024-09-11T05:53:50.577197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:36:07.318555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7252,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984654+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984654+0000","last_peered":"2024-09-11T05:53:50.984654+0000","last_clean":"2024-09-11T05:53:50.984654+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984654+0000","last_undegraded":"2024-09-11T05:53:50.984654+0000","last_fullsized":"2024-09-11T05:53:50.984654+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:19:38.858978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7404,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577512+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577512+0000","last_peered":"2024-09-11T05:53:50.577512+0000","last_clean":"2024-09-11T05:53:50.577512+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577512+0000","last_undegraded":"2024-09-11T05:53:50.577512+0000","last_fullsized":"2024-09-11T05:53:50.577512+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:27.660765+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321073+0000","last_change":"2024-09-11T05:53:35.555476+0000","last_active":"2024-09-11T05:53:51.321073+0000","last_peered":"2024-09-11T05:53:51.321073+0000","last_clean":"2024-09-11T05:53:51.321073+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:51.321073+0000","last_undegraded":"2024-09-11T05:53:51.321073+0000","last_fullsized":"2024-09-11T05:53:51.321073+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:48:33.979145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6221,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577619+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577619+0000","last_peered":"2024-09-11T05:53:50.577619+0000","last_clean":"2024-09-11T05:53:50.577619+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577619+0000","last_undegraded":"2024-09-11T05:53:50.577619+0000","last_fullsized":"2024-09-11T05:53:50.577619+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:38:32.189314+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3677,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578121+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578121+0000","last_peered":"2024-09-11T05:53:50.578121+0000","last_clean":"2024-09-11T05:53:50.578121+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578121+0000","last_undegraded":"2024-09-11T05:53:50.578121+0000","last_fullsized":"2024-09-11T05:53:50.578121+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:14:57.790323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8007,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577878+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577878+0000","last_peered":"2024-09-11T05:53:50.577878+0000","last_clean":"2024-09-11T05:53:50.577878+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577878+0000","last_undegraded":"2024-09-11T05:53:50.577878+0000","last_fullsized":"2024-09-11T05:53:50.577878+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:03:44.318939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9449,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321147+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321147+0000","last_peered":"2024-09-11T05:53:51.321147+0000","last_clean":"2024-09-11T05:53:51.321147+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321147+0000","last_undegraded":"2024-09-11T05:53:51.321147+0000","last_fullsized":"2024-09-11T05:53:51.321147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:58.857313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4038,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984586+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984586+0000","last_peered":"2024-09-11T05:53:50.984586+0000","last_clean":"2024-09-11T05:53:50.984586+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984586+0000","last_undegraded":"2024-09-11T05:53:50.984586+0000","last_fullsized":"2024-09-11T05:53:50.984586+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:20:03.379238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20120,"ondisk_log_size":20120,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69284,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302556,"statfs":{"total":96636764160,"available":96565817344,"internally_reserved":0,"allocated":2048000,"data_stored":977191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41524,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330316,"statfs":{"total":96636764160,"available":96594243584,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2052096,"data_stored":977210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60136,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311704,"statfs":{"total":96636764160,"available":96575184896,"internally_reserved":0,"allocated":2637824,"data_stored":1567540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:53:55.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:53:55.825 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:53:55.825 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":171,"fsid":"d5628057-c743-43f5-a51d-1915a16ccfdd","created":"2024-09-11T05:50:35.802404+0000","modified":"2024-09-11T05:53:50.563505+0000","last_up_change":"2024-09-11T05:50:49.084592+0000","last_in_change":"2024-09-11T05:53:34.802739+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:50:50.566112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:50:54.170388+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-11T05:51:04.331649+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2fdc5357-7501-4582-b301-f5eede8a7bc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6817","nonce":4049823921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6819","nonce":4049823921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6823","nonce":4049823921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4049823921},{"type":"v1","addr":"172.21.15.165:6821","nonce":4049823921}]},"public_addr":"172.21.15.165:6817/4049823921","cluster_addr":"172.21.15.165:6819/4049823921","heartbeat_back_addr":"172.21.15.165:6823/4049823921","heartbeat_front_addr":"172.21.15.165:6821/4049823921","state":["exists","up"]},{"osd":1,"uuid":"78f30980-40a1-46fd-bc1a-a656b32210a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6801","nonce":2461301418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6803","nonce":2461301418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6807","nonce":2461301418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":2461301418},{"type":"v1","addr":"172.21.15.165:6805","nonce":2461301418}]},"public_addr":"172.21.15.165:6801/2461301418","cluster_addr":"172.21.15.165:6803/2461301418","heartbeat_back_addr":"172.21.15.165:6807/2461301418","heartbeat_front_addr":"172.21.15.165:6805/2461301418","state":["exists","up"]},{"osd":2,"uuid":"2fab9ebc-408e-4bc1-b3ad-385d0e396f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6809","nonce":3166670912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6811","nonce":3166670912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6816","nonce":3166670912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3166670912},{"type":"v1","addr":"172.21.15.165:6813","nonce":3166670912}]},"public_addr":"172.21.15.165:6809/3166670912","cluster_addr":"172.21.15.165:6811/3166670912","heartbeat_back_addr":"172.21.15.165:6816/3166670912","heartbeat_front_addr":"172.21.15.165:6813/3166670912","state":["exists","up"]},{"osd":3,"uuid":"d3115ede-8d17-4a27-8629-40b19e3c7ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6825","nonce":3322140846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6827","nonce":3322140846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6831","nonce":3322140846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":3322140846},{"type":"v1","addr":"172.21.15.165:6829","nonce":3322140846}]},"public_addr":"172.21.15.165:6825/3322140846","cluster_addr":"172.21.15.165:6827/3322140846","heartbeat_back_addr":"172.21.15.165:6831/3322140846","heartbeat_front_addr":"172.21.15.165:6829/3322140846","state":["exists","up"]},{"osd":4,"uuid":"7fa88502-3ffd-4aac-924b-0d6c65737ac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6801","nonce":347599006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6803","nonce":347599006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6807","nonce":347599006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":347599006},{"type":"v1","addr":"172.21.15.172:6805","nonce":347599006}]},"public_addr":"172.21.15.172:6801/347599006","cluster_addr":"172.21.15.172:6803/347599006","heartbeat_back_addr":"172.21.15.172:6807/347599006","heartbeat_front_addr":"172.21.15.172:6805/347599006","state":["exists","up"]},{"osd":5,"uuid":"03238cca-4d2b-412d-9fcd-f489cb4a8455","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6817","nonce":4259095655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6819","nonce":4259095655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6823","nonce":4259095655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":4259095655},{"type":"v1","addr":"172.21.15.172:6821","nonce":4259095655}]},"public_addr":"172.21.15.172:6817/4259095655","cluster_addr":"172.21.15.172:6819/4259095655","heartbeat_back_addr":"172.21.15.172:6823/4259095655","heartbeat_front_addr":"172.21.15.172:6821/4259095655","state":["exists","up"]},{"osd":6,"uuid":"6c46c2d3-7dd1-4b3a-ad76-391c6e624b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6825","nonce":2366978310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6827","nonce":2366978310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6831","nonce":2366978310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2366978310},{"type":"v1","addr":"172.21.15.172:6829","nonce":2366978310}]},"public_addr":"172.21.15.172:6825/2366978310","cluster_addr":"172.21.15.172:6827/2366978310","heartbeat_back_addr":"172.21.15.172:6831/2366978310","heartbeat_front_addr":"172.21.15.172:6829/2366978310","state":["exists","up"]},{"osd":7,"uuid":"f849014a-7b82-4361-8a23-3b7a0be5f5c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6809","nonce":2576837456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6811","nonce":2576837456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6815","nonce":2576837456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2576837456},{"type":"v1","addr":"172.21.15.172:6813","nonce":2576837456}]},"public_addr":"172.21.15.172:6809/2576837456","cluster_addr":"172.21.15.172:6811/2576837456","heartbeat_back_addr":"172.21.15.172:6815/2576837456","heartbeat_front_addr":"172.21.15.172:6813/2576837456","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.605278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.681171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.531762+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:46.158901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.412535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.537269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.433446+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:50:45.612887+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:53:56.843 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T05:53:56.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:57.005 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:57.005 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:57.005 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:57.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T05:53:57.294 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T05:53:57.312 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T05:53:57.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:57.459 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:57.459 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:57.459 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:57.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T05:53:57.753 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T05:53:57.770 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T05:53:57.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:57.903 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:57.904 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:57.904 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:57.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T05:53:58.197 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T05:53:58.215 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T05:53:58.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:58.356 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:58.356 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:58.356 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:58.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T05:53:58.646 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T05:53:58.664 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T05:53:58.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:58.803 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:58.803 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:58.803 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:58.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T05:53:59.096 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T05:53:59.115 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T05:53:59.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:59.255 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:59.255 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:59.255 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:59.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T05:53:59.548 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T05:53:59.568 INFO:tasks.ceph:Scrubbing osd.6 2024-09-11T05:53:59.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:53:59.703 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:53:59.703 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:53:59.703 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:53:59.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-11T05:53:59.995 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 6 to deep-scrub 2024-09-11T05:54:00.013 INFO:tasks.ceph:Scrubbing osd.7 2024-09-11T05:54:00.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:54:00.149 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-11T05:54:00.149 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:54:00.150 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-11T05:54:00.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-11T05:54:00.441 INFO:teuthology.orchestra.run.smithi165.stderr:instructed osd(s) 7 to deep-scrub 2024-09-11T05:54:00.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:54:00.753 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:54:00.754 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:54:00.771 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-11T05:54:00.540889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20278,"ondisk_log_size":20278,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426344,"kb_used_data":17240,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548376,"statfs":{"total":773094113280,"available":772657537024,"internally_reserved":0,"allocated":17653760,"data_stored":9044692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003024"},"pg_stats":[{"pgid":"3.1f","version":"154'672","reported_seq":3765,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984607+0000","last_change":"2024-09-11T05:53:42.564009+0000","last_active":"2024-09-11T05:53:50.984607+0000","last_peered":"2024-09-11T05:53:50.984607+0000","last_clean":"2024-09-11T05:53:50.984607+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:53:50.984607+0000","last_undegraded":"2024-09-11T05:53:50.984607+0000","last_fullsized":"2024-09-11T05:53:50.984607+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"151'875","reported_seq":3804,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578406+0000","last_change":"2024-09-11T05:53:32.546408+0000","last_active":"2024-09-11T05:53:50.578406+0000","last_peered":"2024-09-11T05:53:50.578406+0000","last_clean":"2024-09-11T05:53:50.578406+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:50.578406+0000","last_undegraded":"2024-09-11T05:53:50.578406+0000","last_fullsized":"2024-09-11T05:53:50.578406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"149'300","reported_seq":3666,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321347+0000","last_change":"2024-09-11T05:53:39.866214+0000","last_active":"2024-09-11T05:53:51.321347+0000","last_peered":"2024-09-11T05:53:51.321347+0000","last_clean":"2024-09-11T05:53:51.321347+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:53:51.321347+0000","last_undegraded":"2024-09-11T05:53:51.321347+0000","last_fullsized":"2024-09-11T05:53:51.321347+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"154'531","reported_seq":4075,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321241+0000","last_change":"2024-09-11T05:53:41.551684+0000","last_active":"2024-09-11T05:53:51.321241+0000","last_peered":"2024-09-11T05:53:51.321241+0000","last_clean":"2024-09-11T05:53:51.321241+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:53:51.321241+0000","last_undegraded":"2024-09-11T05:53:51.321241+0000","last_fullsized":"2024-09-11T05:53:51.321241+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"171'581","reported_seq":7993,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.166537+0000","last_change":"2024-09-11T05:53:59.166645+0000","last_active":"2024-09-11T05:53:59.166537+0000","last_peered":"2024-09-11T05:53:59.166537+0000","last_clean":"2024-09-11T05:53:59.166537+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:59.166537+0000","last_undegraded":"2024-09-11T05:53:59.166537+0000","last_fullsized":"2024-09-11T05:53:59.166537+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'581","last_scrub_stamp":"2024-09-11T05:53:59.166447+0000","last_deep_scrub":"171'581","last_deep_scrub_stamp":"2024-09-11T05:53:59.166447+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.166447+0000","objects_scrubbed":6,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:31.770879+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"171'428","reported_seq":8584,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.661046+0000","last_change":"2024-09-11T05:53:59.661142+0000","last_active":"2024-09-11T05:53:59.661046+0000","last_peered":"2024-09-11T05:53:59.661046+0000","last_clean":"2024-09-11T05:53:59.661046+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:59.661046+0000","last_undegraded":"2024-09-11T05:53:59.661046+0000","last_fullsized":"2024-09-11T05:53:59.661046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'428","last_scrub_stamp":"2024-09-11T05:53:59.660861+0000","last_deep_scrub":"171'428","last_deep_scrub_stamp":"2024-09-11T05:53:59.660861+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.660861+0000","objects_scrubbed":15,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:17:58.498841+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"153'704","reported_seq":4748,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392162+0000","last_change":"2024-09-11T05:53:32.543874+0000","last_active":"2024-09-11T05:53:51.392162+0000","last_peered":"2024-09-11T05:53:51.392162+0000","last_clean":"2024-09-11T05:53:51.392162+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:53:51.392162+0000","last_undegraded":"2024-09-11T05:53:51.392162+0000","last_fullsized":"2024-09-11T05:53:51.392162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'496","reported_seq":8079,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577516+0000","last_change":"2024-09-11T05:53:43.563133+0000","last_active":"2024-09-11T05:53:50.577516+0000","last_peered":"2024-09-11T05:53:50.577516+0000","last_clean":"2024-09-11T05:53:50.577516+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:53:50.577516+0000","last_undegraded":"2024-09-11T05:53:50.577516+0000","last_fullsized":"2024-09-11T05:53:50.577516+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'463","reported_seq":4560,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984598+0000","last_change":"2024-09-11T05:53:40.548466+0000","last_active":"2024-09-11T05:53:50.984598+0000","last_peered":"2024-09-11T05:53:50.984598+0000","last_clean":"2024-09-11T05:53:50.984598+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:53:50.984598+0000","last_undegraded":"2024-09-11T05:53:50.984598+0000","last_fullsized":"2024-09-11T05:53:50.984598+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"151'368","reported_seq":9685,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321608+0000","last_change":"2024-09-11T05:53:39.867591+0000","last_active":"2024-09-11T05:53:51.321608+0000","last_peered":"2024-09-11T05:53:51.321608+0000","last_clean":"2024-09-11T05:53:51.321608+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:53:51.321608+0000","last_undegraded":"2024-09-11T05:53:51.321608+0000","last_fullsized":"2024-09-11T05:53:51.321608+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"149'498","reported_seq":7862,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393958+0000","last_change":"2024-09-11T05:53:32.563223+0000","last_active":"2024-09-11T05:53:51.393958+0000","last_peered":"2024-09-11T05:53:51.393958+0000","last_clean":"2024-09-11T05:53:51.393958+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:53:51.393958+0000","last_undegraded":"2024-09-11T05:53:51.393958+0000","last_fullsized":"2024-09-11T05:53:51.393958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:59:57.889332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"153'919","reported_seq":6207,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578414+0000","last_change":"2024-09-11T05:53:32.579334+0000","last_active":"2024-09-11T05:53:50.578414+0000","last_peered":"2024-09-11T05:53:50.578414+0000","last_clean":"2024-09-11T05:53:50.578414+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:53:50.578414+0000","last_undegraded":"2024-09-11T05:53:50.578414+0000","last_fullsized":"2024-09-11T05:53:50.578414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394699+0000","last_change":"2024-09-11T05:52:43.797254+0000","last_active":"2024-09-11T05:53:51.394699+0000","last_peered":"2024-09-11T05:53:51.394699+0000","last_clean":"2024-09-11T05:53:51.394699+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:53:51.394699+0000","last_undegraded":"2024-09-11T05:53:51.394699+0000","last_fullsized":"2024-09-11T05:53:51.394699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:53:33.678447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"171'829","reported_seq":6780,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.167140+0000","last_change":"2024-09-11T05:53:58.167212+0000","last_active":"2024-09-11T05:53:58.167140+0000","last_peered":"2024-09-11T05:53:58.167140+0000","last_clean":"2024-09-11T05:53:58.167140+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:58.167140+0000","last_undegraded":"2024-09-11T05:53:58.167140+0000","last_fullsized":"2024-09-11T05:53:58.167140+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'829","last_scrub_stamp":"2024-09-11T05:53:58.166965+0000","last_deep_scrub":"171'829","last_deep_scrub_stamp":"2024-09-11T05:53:58.166965+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.166965+0000","objects_scrubbed":9,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:37.284005+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":215,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.513972+0000","last_change":"2024-09-11T05:53:58.513994+0000","last_active":"2024-09-11T05:53:58.513972+0000","last_peered":"2024-09-11T05:53:58.513972+0000","last_clean":"2024-09-11T05:53:58.513972+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:58.513972+0000","last_undegraded":"2024-09-11T05:53:58.513972+0000","last_fullsized":"2024-09-11T05:53:58.513972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:53:58.513899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:53:58.513899+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.513899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:16.335044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"152'492","reported_seq":3087,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578649+0000","last_change":"2024-09-11T05:53:43.562412+0000","last_active":"2024-09-11T05:53:50.578649+0000","last_peered":"2024-09-11T05:53:50.578649+0000","last_clean":"2024-09-11T05:53:50.578649+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:53:50.578649+0000","last_undegraded":"2024-09-11T05:53:50.578649+0000","last_fullsized":"2024-09-11T05:53:50.578649+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394291+0000","last_change":"2024-09-11T05:52:49.857747+0000","last_active":"2024-09-11T05:53:51.394291+0000","last_peered":"2024-09-11T05:53:51.394291+0000","last_clean":"2024-09-11T05:53:51.394291+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:53:51.394291+0000","last_undegraded":"2024-09-11T05:53:51.394291+0000","last_fullsized":"2024-09-11T05:53:51.394291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:57.353066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"153'389","reported_seq":6299,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392224+0000","last_change":"2024-09-11T05:53:32.544729+0000","last_active":"2024-09-11T05:53:51.392224+0000","last_peered":"2024-09-11T05:53:51.392224+0000","last_clean":"2024-09-11T05:53:51.392224+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:53:51.392224+0000","last_undegraded":"2024-09-11T05:53:51.392224+0000","last_fullsized":"2024-09-11T05:53:51.392224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":217,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.392305+0000","last_change":"2024-09-11T05:52:49.857184+0000","last_active":"2024-09-11T05:53:51.392305+0000","last_peered":"2024-09-11T05:53:51.392305+0000","last_clean":"2024-09-11T05:53:51.392305+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:53:51.392305+0000","last_undegraded":"2024-09-11T05:53:51.392305+0000","last_fullsized":"2024-09-11T05:53:51.392305+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"154'638","reported_seq":6191,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321580+0000","last_change":"2024-09-11T05:53:40.554349+0000","last_active":"2024-09-11T05:53:51.321580+0000","last_peered":"2024-09-11T05:53:51.321580+0000","last_clean":"2024-09-11T05:53:51.321580+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:53:51.321580+0000","last_undegraded":"2024-09-11T05:53:51.321580+0000","last_fullsized":"2024-09-11T05:53:51.321580+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"154'914","reported_seq":4627,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578300+0000","last_change":"2024-09-11T05:53:32.575896+0000","last_active":"2024-09-11T05:53:50.578300+0000","last_peered":"2024-09-11T05:53:50.578300+0000","last_clean":"2024-09-11T05:53:50.578300+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:53:50.578300+0000","last_undegraded":"2024-09-11T05:53:50.578300+0000","last_fullsized":"2024-09-11T05:53:50.578300+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":45,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321230+0000","last_change":"2024-09-11T05:53:35.560401+0000","last_active":"2024-09-11T05:53:51.321230+0000","last_peered":"2024-09-11T05:53:51.321230+0000","last_clean":"2024-09-11T05:53:51.321230+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:53:51.321230+0000","last_undegraded":"2024-09-11T05:53:51.321230+0000","last_fullsized":"2024-09-11T05:53:51.321230+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"171'678","reported_seq":3535,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.533784+0000","last_change":"2024-09-11T05:53:59.533836+0000","last_active":"2024-09-11T05:53:59.533784+0000","last_peered":"2024-09-11T05:53:59.533784+0000","last_clean":"2024-09-11T05:53:59.533784+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:59.533784+0000","last_undegraded":"2024-09-11T05:53:59.533784+0000","last_fullsized":"2024-09-11T05:53:59.533784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'678","last_scrub_stamp":"2024-09-11T05:53:59.533681+0000","last_deep_scrub":"171'678","last_deep_scrub_stamp":"2024-09-11T05:53:59.533681+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.533681+0000","objects_scrubbed":7,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:53:04.241243+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":326,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.650657+0000","last_change":"2024-09-11T05:53:58.650677+0000","last_active":"2024-09-11T05:53:58.650657+0000","last_peered":"2024-09-11T05:53:58.650657+0000","last_clean":"2024-09-11T05:53:58.650657+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:58.650657+0000","last_undegraded":"2024-09-11T05:53:58.650657+0000","last_fullsized":"2024-09-11T05:53:58.650657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:53:58.650572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:53:58.650572+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.650572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:47.801927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'787","reported_seq":7253,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984685+0000","last_change":"2024-09-11T05:53:32.557223+0000","last_active":"2024-09-11T05:53:50.984685+0000","last_peered":"2024-09-11T05:53:50.984685+0000","last_clean":"2024-09-11T05:53:50.984685+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:53:50.984685+0000","last_undegraded":"2024-09-11T05:53:50.984685+0000","last_fullsized":"2024-09-11T05:53:50.984685+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393821+0000","last_change":"2024-09-11T05:52:05.518757+0000","last_active":"2024-09-11T05:53:51.393821+0000","last_peered":"2024-09-11T05:53:51.393821+0000","last_clean":"2024-09-11T05:53:51.393821+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:53:51.393821+0000","last_undegraded":"2024-09-11T05:53:51.393821+0000","last_fullsized":"2024-09-11T05:53:51.393821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:55.126278+0000","last_clean_scrub_stamp":"2024-09-11T05:50:55.126278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:24.707812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'474","reported_seq":7405,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577647+0000","last_change":"2024-09-11T05:53:43.563625+0000","last_active":"2024-09-11T05:53:50.577647+0000","last_peered":"2024-09-11T05:53:50.577647+0000","last_clean":"2024-09-11T05:53:50.577647+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:53:50.577647+0000","last_undegraded":"2024-09-11T05:53:50.577647+0000","last_fullsized":"2024-09-11T05:53:50.577647+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"171'2","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.607813+0000","last_change":"2024-09-11T05:53:59.607902+0000","last_active":"2024-09-11T05:53:59.607813+0000","last_peered":"2024-09-11T05:53:59.607813+0000","last_clean":"2024-09-11T05:53:59.607813+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:59.607813+0000","last_undegraded":"2024-09-11T05:53:59.607813+0000","last_fullsized":"2024-09-11T05:53:59.607813+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'2","last_scrub_stamp":"2024-09-11T05:53:59.607623+0000","last_deep_scrub":"171'2","last_deep_scrub_stamp":"2024-09-11T05:53:59.607623+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.607623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:46.734546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":494,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394143+0000","last_change":"2024-09-11T05:52:54.929124+0000","last_active":"2024-09-11T05:53:51.394143+0000","last_peered":"2024-09-11T05:53:51.394143+0000","last_clean":"2024-09-11T05:53:51.394143+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:53:51.394143+0000","last_undegraded":"2024-09-11T05:53:51.394143+0000","last_fullsized":"2024-09-11T05:53:51.394143+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:50:51.090339+0000","last_clean_scrub_stamp":"2024-09-11T05:50:51.090339+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:33.851999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'559","reported_seq":6222,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577724+0000","last_change":"2024-09-11T05:53:43.561147+0000","last_active":"2024-09-11T05:53:50.577724+0000","last_peered":"2024-09-11T05:53:50.577724+0000","last_clean":"2024-09-11T05:53:50.577724+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:53:50.577724+0000","last_undegraded":"2024-09-11T05:53:50.577724+0000","last_fullsized":"2024-09-11T05:53:50.577724+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'317","reported_seq":3678,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.578250+0000","last_change":"2024-09-11T05:53:32.576888+0000","last_active":"2024-09-11T05:53:50.578250+0000","last_peered":"2024-09-11T05:53:50.578250+0000","last_clean":"2024-09-11T05:53:50.578250+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:53:50.578250+0000","last_undegraded":"2024-09-11T05:53:50.578250+0000","last_fullsized":"2024-09-11T05:53:50.578250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"153'762","reported_seq":6530,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394637+0000","last_change":"2024-09-11T05:53:49.572387+0000","last_active":"2024-09-11T05:53:51.394637+0000","last_peered":"2024-09-11T05:53:51.394637+0000","last_clean":"2024-09-11T05:53:51.394637+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:53:51.394637+0000","last_undegraded":"2024-09-11T05:53:51.394637+0000","last_fullsized":"2024-09-11T05:53:51.394637+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:06.728858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"154'533","reported_seq":8008,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.577944+0000","last_change":"2024-09-11T05:53:43.565726+0000","last_active":"2024-09-11T05:53:50.577944+0000","last_peered":"2024-09-11T05:53:50.577944+0000","last_clean":"2024-09-11T05:53:50.577944+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:53:50.577944+0000","last_undegraded":"2024-09-11T05:53:50.577944+0000","last_fullsized":"2024-09-11T05:53:50.577944+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"146'445","reported_seq":9450,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.321262+0000","last_change":"2024-09-11T05:53:41.556202+0000","last_active":"2024-09-11T05:53:51.321262+0000","last_peered":"2024-09-11T05:53:51.321262+0000","last_clean":"2024-09-11T05:53:51.321262+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:53:51.321262+0000","last_undegraded":"2024-09-11T05:53:51.321262+0000","last_fullsized":"2024-09-11T05:53:51.321262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"151'563","reported_seq":5953,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394031+0000","last_change":"2024-09-11T05:53:40.558210+0000","last_active":"2024-09-11T05:53:51.394031+0000","last_peered":"2024-09-11T05:53:51.394031+0000","last_clean":"2024-09-11T05:53:51.394031+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:53:51.394031+0000","last_undegraded":"2024-09-11T05:53:51.394031+0000","last_fullsized":"2024-09-11T05:53:51.394031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:36.449853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"152'1027","reported_seq":12874,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394412+0000","last_change":"2024-09-11T05:53:32.565027+0000","last_active":"2024-09-11T05:53:51.394412+0000","last_peered":"2024-09-11T05:53:51.394412+0000","last_clean":"2024-09-11T05:53:51.394412+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:53:51.394412+0000","last_undegraded":"2024-09-11T05:53:51.394412+0000","last_fullsized":"2024-09-11T05:53:51.394412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:54.369808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'940","reported_seq":3878,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394393+0000","last_change":"2024-09-11T05:53:32.568303+0000","last_active":"2024-09-11T05:53:51.394393+0000","last_peered":"2024-09-11T05:53:51.394393+0000","last_clean":"2024-09-11T05:53:51.394393+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:53:51.394393+0000","last_undegraded":"2024-09-11T05:53:51.394393+0000","last_fullsized":"2024-09-11T05:53:51.394393+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:07:53.127242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"151'1171","reported_seq":4164,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.394597+0000","last_change":"2024-09-11T05:53:32.544739+0000","last_active":"2024-09-11T05:53:51.394597+0000","last_peered":"2024-09-11T05:53:51.394597+0000","last_clean":"2024-09-11T05:53:51.394597+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:53:51.394597+0000","last_undegraded":"2024-09-11T05:53:51.394597+0000","last_fullsized":"2024-09-11T05:53:51.394597+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:59.233723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'592","reported_seq":5040,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393837+0000","last_change":"2024-09-11T05:53:32.562959+0000","last_active":"2024-09-11T05:53:51.393837+0000","last_peered":"2024-09-11T05:53:51.393837+0000","last_clean":"2024-09-11T05:53:51.393837+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:53:51.393837+0000","last_undegraded":"2024-09-11T05:53:51.393837+0000","last_fullsized":"2024-09-11T05:53:51.393837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:45.748139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"154'741","reported_seq":4039,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:50.984643+0000","last_change":"2024-09-11T05:53:32.557779+0000","last_active":"2024-09-11T05:53:50.984643+0000","last_peered":"2024-09-11T05:53:50.984643+0000","last_clean":"2024-09-11T05:53:50.984643+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:53:50.984643+0000","last_undegraded":"2024-09-11T05:53:50.984643+0000","last_fullsized":"2024-09-11T05:53:50.984643+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'471","reported_seq":14381,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:51.393918+0000","last_change":"2024-09-11T05:53:32.564331+0000","last_active":"2024-09-11T05:53:51.393918+0000","last_peered":"2024-09-11T05:53:51.393918+0000","last_clean":"2024-09-11T05:53:51.393918+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:53:51.393918+0000","last_undegraded":"2024-09-11T05:53:51.393918+0000","last_fullsized":"2024-09-11T05:53:51.393918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:51:05.192985+0000","last_clean_scrub_stamp":"2024-09-11T05:51:05.192985+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:27:59.377138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20157,"ondisk_log_size":20157,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542187,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78192,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293648,"statfs":{"total":96636764160,"available":96556695552,"internally_reserved":0,"allocated":2650112,"data_stored":1573353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542188,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2060288,"data_stored":983004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542188,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69292,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302548,"statfs":{"total":96636764160,"available":96565809152,"internally_reserved":0,"allocated":2060288,"data_stored":983004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59564,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312276,"statfs":{"total":96636764160,"available":96575770624,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41512,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330328,"statfs":{"total":96636764160,"available":96594255872,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2064384,"data_stored":983023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542188,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60140,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311700,"statfs":{"total":96636764160,"available":96575180800,"internally_reserved":0,"allocated":2650112,"data_stored":1573353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:54:00.776 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.777 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T05:50:55.126278+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T05:50:55.126278+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T05:50:55.126278+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T05:50:55.126278+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.778 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T05:50:55.126278+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T05:50:51.090339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=50, tm_sec=51, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.779 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-11T05:51:05.192985+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=51, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=53, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:54:00.780 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T05:54:20.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:54:21.095 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-11T05:54:21.096 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-09-11T05:54:21.114 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-11T05:54:20.547275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":283,"num_object_clones":0,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":275,"num_read":220339,"num_read_kb":578469,"num_write":24220,"num_write_kb":305693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4781,"num_bytes_recovered":141292172,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20522,"ondisk_log_size":20522,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426328,"kb_used_data":17240,"kb_used_omap":12,"kb_used_meta":408947,"kb_avail":754548392,"statfs":{"total":773094113280,"available":772657553408,"internally_reserved":0,"allocated":17653760,"data_stored":9044692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":418762290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003839"},"pg_stats":[{"pgid":"3.1f","version":"171'677","reported_seq":3778,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:03.202881+0000","last_change":"2024-09-11T05:54:03.202950+0000","last_active":"2024-09-11T05:54:03.202881+0000","last_peered":"2024-09-11T05:54:03.202881+0000","last_clean":"2024-09-11T05:54:03.202881+0000","last_became_active":"2024-09-11T05:53:42.560004+0000","last_became_peered":"2024-09-11T05:53:42.560004+0000","last_unstale":"2024-09-11T05:54:03.202881+0000","last_undegraded":"2024-09-11T05:54:03.202881+0000","last_fullsized":"2024-09-11T05:54:03.202881+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'677","last_scrub_stamp":"2024-09-11T05:54:03.202668+0000","last_deep_scrub":"171'677","last_deep_scrub_stamp":"2024-09-11T05:54:03.202668+0000","last_clean_scrub_stamp":"2024-09-11T05:54:03.202668+0000","objects_scrubbed":5,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:33:14.114272+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0034630799999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":3499,"num_read_kb":8178,"num_write":766,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":7945632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"171'887","reported_seq":3824,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.563306+0000","last_change":"2024-09-11T05:53:59.563383+0000","last_active":"2024-09-11T05:53:59.563306+0000","last_peered":"2024-09-11T05:53:59.563306+0000","last_clean":"2024-09-11T05:53:59.563306+0000","last_became_active":"2024-09-11T05:51:05.514222+0000","last_became_peered":"2024-09-11T05:51:05.514222+0000","last_unstale":"2024-09-11T05:53:59.563306+0000","last_undegraded":"2024-09-11T05:53:59.563306+0000","last_fullsized":"2024-09-11T05:53:59.563306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'887","last_scrub_stamp":"2024-09-11T05:53:59.563205+0000","last_deep_scrub":"171'887","last_deep_scrub_stamp":"2024-09-11T05:53:59.563205+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.563205+0000","objects_scrubbed":12,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:05:56.840587+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0013529900000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":6604,"num_read_kb":13063,"num_write":1285,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"171'310","reported_seq":3684,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.599245+0000","last_change":"2024-09-11T05:54:01.599329+0000","last_active":"2024-09-11T05:54:01.599245+0000","last_peered":"2024-09-11T05:54:01.599245+0000","last_clean":"2024-09-11T05:54:01.599245+0000","last_became_active":"2024-09-11T05:53:39.859500+0000","last_became_peered":"2024-09-11T05:53:39.859500+0000","last_unstale":"2024-09-11T05:54:01.599245+0000","last_undegraded":"2024-09-11T05:54:01.599245+0000","last_fullsized":"2024-09-11T05:54:01.599245+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'310","last_scrub_stamp":"2024-09-11T05:54:01.599017+0000","last_deep_scrub":"171'310","last_deep_scrub_stamp":"2024-09-11T05:54:01.599017+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.599017+0000","objects_scrubbed":10,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:04:08.471343+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0023628529999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2754,"num_read_kb":13899,"num_write":398,"num_write_kb":7472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":5470402,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"171'534","reported_seq":4086,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:04.602533+0000","last_change":"2024-09-11T05:54:04.602601+0000","last_active":"2024-09-11T05:54:04.602533+0000","last_peered":"2024-09-11T05:54:04.602533+0000","last_clean":"2024-09-11T05:54:04.602533+0000","last_became_active":"2024-09-11T05:53:41.547440+0000","last_became_peered":"2024-09-11T05:53:41.547440+0000","last_unstale":"2024-09-11T05:54:04.602533+0000","last_undegraded":"2024-09-11T05:54:04.602533+0000","last_fullsized":"2024-09-11T05:54:04.602533+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'534","last_scrub_stamp":"2024-09-11T05:54:04.602348+0000","last_deep_scrub":"171'534","last_deep_scrub_stamp":"2024-09-11T05:54:04.602348+0000","last_clean_scrub_stamp":"2024-09-11T05:54:04.602348+0000","objects_scrubbed":3,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:15:38.141474+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0039909100000000003,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":4319,"num_read_kb":8873,"num_write":664,"num_write_kb":5228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1872467,"num_keys_recovered":69,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"171'581","reported_seq":7993,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.166537+0000","last_change":"2024-09-11T05:53:59.166645+0000","last_active":"2024-09-11T05:53:59.166537+0000","last_peered":"2024-09-11T05:53:59.166537+0000","last_clean":"2024-09-11T05:53:59.166537+0000","last_became_active":"2024-09-11T05:52:54.922877+0000","last_became_peered":"2024-09-11T05:52:54.922877+0000","last_unstale":"2024-09-11T05:53:59.166537+0000","last_undegraded":"2024-09-11T05:53:59.166537+0000","last_fullsized":"2024-09-11T05:53:59.166537+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'581","last_scrub_stamp":"2024-09-11T05:53:59.166447+0000","last_deep_scrub":"171'581","last_deep_scrub_stamp":"2024-09-11T05:53:59.166447+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.166447+0000","objects_scrubbed":6,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:31.770879+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0027665390000000002,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":8197,"num_read_kb":36662,"num_write":679,"num_write_kb":10799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":5155306,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"171'428","reported_seq":8584,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.661046+0000","last_change":"2024-09-11T05:53:59.661142+0000","last_active":"2024-09-11T05:53:59.661046+0000","last_peered":"2024-09-11T05:53:59.661046+0000","last_clean":"2024-09-11T05:53:59.661046+0000","last_became_active":"2024-09-11T05:51:05.512329+0000","last_became_peered":"2024-09-11T05:51:05.512329+0000","last_unstale":"2024-09-11T05:53:59.661046+0000","last_undegraded":"2024-09-11T05:53:59.661046+0000","last_fullsized":"2024-09-11T05:53:59.661046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'428","last_scrub_stamp":"2024-09-11T05:53:59.660861+0000","last_deep_scrub":"171'428","last_deep_scrub_stamp":"2024-09-11T05:53:59.660861+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.660861+0000","objects_scrubbed":15,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:17:58.498841+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0055796320000000002,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":14,"num_read":7845,"num_read_kb":13752,"num_write":469,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"171'715","reported_seq":4767,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.593155+0000","last_change":"2024-09-11T05:54:01.593335+0000","last_active":"2024-09-11T05:54:01.593155+0000","last_peered":"2024-09-11T05:54:01.593155+0000","last_clean":"2024-09-11T05:54:01.593155+0000","last_became_active":"2024-09-11T05:52:08.533616+0000","last_became_peered":"2024-09-11T05:52:08.533616+0000","last_unstale":"2024-09-11T05:54:01.593155+0000","last_undegraded":"2024-09-11T05:54:01.593155+0000","last_fullsized":"2024-09-11T05:54:01.593155+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'715","last_scrub_stamp":"2024-09-11T05:54:01.592918+0000","last_deep_scrub":"171'715","last_deep_scrub_stamp":"2024-09-11T05:54:01.592918+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.592918+0000","objects_scrubbed":11,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:42:12.670437+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.001197893,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5474,"num_read_kb":10501,"num_write":885,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":4832334,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"171'498","reported_seq":8089,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.434369+0000","last_change":"2024-09-11T05:54:02.434432+0000","last_active":"2024-09-11T05:54:02.434369+0000","last_peered":"2024-09-11T05:54:02.434369+0000","last_clean":"2024-09-11T05:54:02.434369+0000","last_became_active":"2024-09-11T05:53:43.560788+0000","last_became_peered":"2024-09-11T05:53:43.560788+0000","last_unstale":"2024-09-11T05:54:02.434369+0000","last_undegraded":"2024-09-11T05:54:02.434369+0000","last_fullsized":"2024-09-11T05:54:02.434369+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'498","last_scrub_stamp":"2024-09-11T05:54:02.434182+0000","last_deep_scrub":"171'498","last_deep_scrub_stamp":"2024-09-11T05:54:02.434182+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.434182+0000","objects_scrubbed":2,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:37.240628+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.011347902999999999,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":8401,"num_read_kb":28604,"num_write":496,"num_write_kb":11137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":7525106,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"171'470","reported_seq":4575,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.178638+0000","last_change":"2024-09-11T05:54:02.178723+0000","last_active":"2024-09-11T05:54:02.178638+0000","last_peered":"2024-09-11T05:54:02.178638+0000","last_clean":"2024-09-11T05:54:02.178638+0000","last_became_active":"2024-09-11T05:53:40.546482+0000","last_became_peered":"2024-09-11T05:53:40.546482+0000","last_unstale":"2024-09-11T05:54:02.178638+0000","last_undegraded":"2024-09-11T05:54:02.178638+0000","last_fullsized":"2024-09-11T05:54:02.178638+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'470","last_scrub_stamp":"2024-09-11T05:54:02.178418+0000","last_deep_scrub":"171'470","last_deep_scrub_stamp":"2024-09-11T05:54:02.178418+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.178418+0000","objects_scrubbed":7,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:59.783128+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0030678469999999998,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4841,"num_read_kb":10224,"num_write":511,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":4453037,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"171'379","reported_seq":9704,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.595044+0000","last_change":"2024-09-11T05:54:02.595166+0000","last_active":"2024-09-11T05:54:02.595044+0000","last_peered":"2024-09-11T05:54:02.595044+0000","last_clean":"2024-09-11T05:54:02.595044+0000","last_became_active":"2024-09-11T05:53:39.860566+0000","last_became_peered":"2024-09-11T05:53:39.860566+0000","last_unstale":"2024-09-11T05:54:02.595044+0000","last_undegraded":"2024-09-11T05:54:02.595044+0000","last_fullsized":"2024-09-11T05:54:02.595044+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'379","last_scrub_stamp":"2024-09-11T05:54:02.594826+0000","last_deep_scrub":"171'379","last_deep_scrub_stamp":"2024-09-11T05:54:02.594826+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.594826+0000","objects_scrubbed":11,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:54:01.353984+0000","scrub_duration":19,"objects_trimmed":2,"snaptrim_duration":0.011326466,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":8741,"num_read_kb":27123,"num_write":391,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1746659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"171'508","reported_seq":7881,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.356380+0000","last_change":"2024-09-11T05:54:01.356460+0000","last_active":"2024-09-11T05:54:01.356380+0000","last_peered":"2024-09-11T05:54:01.356380+0000","last_clean":"2024-09-11T05:54:01.356380+0000","last_became_active":"2024-09-11T05:51:05.510784+0000","last_became_peered":"2024-09-11T05:51:05.510784+0000","last_unstale":"2024-09-11T05:54:01.356380+0000","last_undegraded":"2024-09-11T05:54:01.356380+0000","last_fullsized":"2024-09-11T05:54:01.356380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'508","last_scrub_stamp":"2024-09-11T05:54:01.356180+0000","last_deep_scrub":"171'508","last_deep_scrub_stamp":"2024-09-11T05:54:01.356180+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.356180+0000","objects_scrubbed":10,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:46:33.579884+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0019048299999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8228,"num_read_kb":10445,"num_write":466,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"171'924","reported_seq":6220,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.622626+0000","last_change":"2024-09-11T05:54:00.622713+0000","last_active":"2024-09-11T05:54:00.622626+0000","last_peered":"2024-09-11T05:54:00.622626+0000","last_clean":"2024-09-11T05:54:00.622626+0000","last_became_active":"2024-09-11T05:51:05.518614+0000","last_became_peered":"2024-09-11T05:51:05.518614+0000","last_unstale":"2024-09-11T05:54:00.622626+0000","last_undegraded":"2024-09-11T05:54:00.622626+0000","last_fullsized":"2024-09-11T05:54:00.622626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'924","last_scrub_stamp":"2024-09-11T05:54:00.622428+0000","last_deep_scrub":"171'924","last_deep_scrub_stamp":"2024-09-11T05:54:00.622428+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.622428+0000","objects_scrubbed":5,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:18.435113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0074444690000000004,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":7538,"num_read_kb":23263,"num_write":1099,"num_write_kb":14063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":180,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.499606+0000","last_change":"2024-09-11T05:54:00.499644+0000","last_active":"2024-09-11T05:54:00.499606+0000","last_peered":"2024-09-11T05:54:00.499606+0000","last_clean":"2024-09-11T05:54:00.499606+0000","last_became_active":"2024-09-11T05:52:43.796568+0000","last_became_peered":"2024-09-11T05:52:43.796568+0000","last_unstale":"2024-09-11T05:54:00.499606+0000","last_undegraded":"2024-09-11T05:54:00.499606+0000","last_fullsized":"2024-09-11T05:54:00.499606+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:54:00.499489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:54:00.499489+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.499489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:54:00.108413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"171'829","reported_seq":6780,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.167140+0000","last_change":"2024-09-11T05:53:58.167212+0000","last_active":"2024-09-11T05:53:58.167140+0000","last_peered":"2024-09-11T05:53:58.167140+0000","last_clean":"2024-09-11T05:53:58.167140+0000","last_became_active":"2024-09-11T05:52:52.905753+0000","last_became_peered":"2024-09-11T05:52:52.905753+0000","last_unstale":"2024-09-11T05:53:58.167140+0000","last_undegraded":"2024-09-11T05:53:58.167140+0000","last_fullsized":"2024-09-11T05:53:58.167140+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'829","last_scrub_stamp":"2024-09-11T05:53:58.166965+0000","last_deep_scrub":"171'829","last_deep_scrub_stamp":"2024-09-11T05:53:58.166965+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.166965+0000","objects_scrubbed":9,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:37.284005+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0026263839999999998,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":7257,"num_read_kb":15632,"num_write":959,"num_write_kb":7052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":10933217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":216,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.513994+0000","last_change":"2024-09-11T05:53:58.513994+0000","last_active":"2024-09-11T05:53:58.513994+0000","last_peered":"2024-09-11T05:53:58.513994+0000","last_clean":"2024-09-11T05:53:58.513994+0000","last_became_active":"2024-09-11T05:52:05.513311+0000","last_became_peered":"2024-09-11T05:52:05.513311+0000","last_unstale":"2024-09-11T05:53:58.513994+0000","last_undegraded":"2024-09-11T05:53:58.513994+0000","last_fullsized":"2024-09-11T05:53:58.513994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:53:58.513899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:53:58.513899+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.513899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:16.335044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091330199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"171'502","reported_seq":3105,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.494976+0000","last_change":"2024-09-11T05:54:01.495073+0000","last_active":"2024-09-11T05:54:01.494976+0000","last_peered":"2024-09-11T05:54:01.494976+0000","last_clean":"2024-09-11T05:54:01.494976+0000","last_became_active":"2024-09-11T05:53:43.560821+0000","last_became_peered":"2024-09-11T05:53:43.560821+0000","last_unstale":"2024-09-11T05:54:01.494976+0000","last_undegraded":"2024-09-11T05:54:01.494976+0000","last_fullsized":"2024-09-11T05:54:01.494976+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'502","last_scrub_stamp":"2024-09-11T05:54:01.494788+0000","last_deep_scrub":"171'502","last_deep_scrub_stamp":"2024-09-11T05:54:01.494788+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.494788+0000","objects_scrubbed":10,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:40:00.784287+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.006638812,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3217,"num_read_kb":9013,"num_write":652,"num_write_kb":4859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":336,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:07.377672+0000","last_change":"2024-09-11T05:54:07.377726+0000","last_active":"2024-09-11T05:54:07.377672+0000","last_peered":"2024-09-11T05:54:07.377672+0000","last_clean":"2024-09-11T05:54:07.377672+0000","last_became_active":"2024-09-11T05:52:49.857016+0000","last_became_peered":"2024-09-11T05:52:49.857016+0000","last_unstale":"2024-09-11T05:54:07.377672+0000","last_undegraded":"2024-09-11T05:54:07.377672+0000","last_fullsized":"2024-09-11T05:54:07.377672+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:54:07.377475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:54:07.377475+0000","last_clean_scrub_stamp":"2024-09-11T05:54:07.377475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:09:24.899200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012381899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"171'393","reported_seq":6312,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.558245+0000","last_change":"2024-09-11T05:54:00.558245+0000","last_active":"2024-09-11T05:54:00.558245+0000","last_peered":"2024-09-11T05:54:00.558245+0000","last_clean":"2024-09-11T05:54:00.558245+0000","last_became_active":"2024-09-11T05:52:08.529138+0000","last_became_peered":"2024-09-11T05:52:08.529138+0000","last_unstale":"2024-09-11T05:54:00.558245+0000","last_undegraded":"2024-09-11T05:54:00.558245+0000","last_fullsized":"2024-09-11T05:54:00.558245+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'393","last_scrub_stamp":"2024-09-11T05:54:00.557972+0000","last_deep_scrub":"171'393","last_deep_scrub_stamp":"2024-09-11T05:54:00.557972+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.557972+0000","objects_scrubbed":4,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:12:39.152889+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0015930930000000001,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":6343,"num_read_kb":24559,"num_write":367,"num_write_kb":6601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":2994718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.541479+0000","last_change":"2024-09-11T05:54:02.541479+0000","last_active":"2024-09-11T05:54:02.541479+0000","last_peered":"2024-09-11T05:54:02.541479+0000","last_clean":"2024-09-11T05:54:02.541479+0000","last_became_active":"2024-09-11T05:52:49.856803+0000","last_became_peered":"2024-09-11T05:52:49.856803+0000","last_unstale":"2024-09-11T05:54:02.541479+0000","last_undegraded":"2024-09-11T05:54:02.541479+0000","last_fullsized":"2024-09-11T05:54:02.541479+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:54:02.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:54:02.541155+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.541155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:25:04.105063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"171'648","reported_seq":6209,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:03.576609+0000","last_change":"2024-09-11T05:54:03.576694+0000","last_active":"2024-09-11T05:54:03.576609+0000","last_peered":"2024-09-11T05:54:03.576609+0000","last_clean":"2024-09-11T05:54:03.576609+0000","last_became_active":"2024-09-11T05:53:40.547989+0000","last_became_peered":"2024-09-11T05:53:40.547989+0000","last_unstale":"2024-09-11T05:54:03.576609+0000","last_undegraded":"2024-09-11T05:54:03.576609+0000","last_fullsized":"2024-09-11T05:54:03.576609+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'648","last_scrub_stamp":"2024-09-11T05:54:03.576418+0000","last_deep_scrub":"171'648","last_deep_scrub_stamp":"2024-09-11T05:54:03.576418+0000","last_clean_scrub_stamp":"2024-09-11T05:54:03.576418+0000","objects_scrubbed":10,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:14:10.902831+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0023770509999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8274,"num_read_kb":21823,"num_write":625,"num_write_kb":10526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":4111315,"num_keys_recovered":21,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"171'924","reported_seq":4645,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.652437+0000","last_change":"2024-09-11T05:54:02.652526+0000","last_active":"2024-09-11T05:54:02.652437+0000","last_peered":"2024-09-11T05:54:02.652437+0000","last_clean":"2024-09-11T05:54:02.652437+0000","last_became_active":"2024-09-11T05:52:56.530660+0000","last_became_peered":"2024-09-11T05:52:56.530660+0000","last_unstale":"2024-09-11T05:54:02.652437+0000","last_undegraded":"2024-09-11T05:54:02.652437+0000","last_fullsized":"2024-09-11T05:54:02.652437+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'924","last_scrub_stamp":"2024-09-11T05:54:02.652254+0000","last_deep_scrub":"171'924","last_deep_scrub_stamp":"2024-09-11T05:54:02.652254+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.652254+0000","objects_scrubbed":11,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:04.713646+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.004457567,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":5253,"num_read_kb":10317,"num_write":1166,"num_write_kb":6269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":4362210,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"171'3","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.611075+0000","last_change":"2024-09-11T05:54:00.611189+0000","last_active":"2024-09-11T05:54:00.611075+0000","last_peered":"2024-09-11T05:54:00.611075+0000","last_clean":"2024-09-11T05:54:00.611075+0000","last_became_active":"2024-09-11T05:53:35.550921+0000","last_became_peered":"2024-09-11T05:53:35.550921+0000","last_unstale":"2024-09-11T05:54:00.611075+0000","last_undegraded":"2024-09-11T05:54:00.611075+0000","last_fullsized":"2024-09-11T05:54:00.611075+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'3","last_scrub_stamp":"2024-09-11T05:54:00.610774+0000","last_deep_scrub":"171'3","last_deep_scrub_stamp":"2024-09-11T05:54:00.610774+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.610774+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:20:14.672424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"171'678","reported_seq":3536,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.533836+0000","last_change":"2024-09-11T05:53:59.533836+0000","last_active":"2024-09-11T05:53:59.533836+0000","last_peered":"2024-09-11T05:53:59.533836+0000","last_clean":"2024-09-11T05:53:59.533836+0000","last_became_active":"2024-09-11T05:52:08.527981+0000","last_became_peered":"2024-09-11T05:52:08.527981+0000","last_unstale":"2024-09-11T05:53:59.533836+0000","last_undegraded":"2024-09-11T05:53:59.533836+0000","last_fullsized":"2024-09-11T05:53:59.533836+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'678","last_scrub_stamp":"2024-09-11T05:53:59.533681+0000","last_deep_scrub":"171'678","last_deep_scrub_stamp":"2024-09-11T05:53:59.533681+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.533681+0000","objects_scrubbed":7,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:53:04.241243+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0013963389999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4487,"num_read_kb":10175,"num_write":701,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":5628856,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":326,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:58.650657+0000","last_change":"2024-09-11T05:53:58.650677+0000","last_active":"2024-09-11T05:53:58.650657+0000","last_peered":"2024-09-11T05:53:58.650657+0000","last_clean":"2024-09-11T05:53:58.650657+0000","last_became_active":"2024-09-11T05:50:56.150929+0000","last_became_peered":"2024-09-11T05:50:56.150929+0000","last_unstale":"2024-09-11T05:53:58.650657+0000","last_undegraded":"2024-09-11T05:53:58.650657+0000","last_fullsized":"2024-09-11T05:53:58.650657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:53:58.650572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:53:58.650572+0000","last_clean_scrub_stamp":"2024-09-11T05:53:58.650572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:47.801927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011372979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"171'801","reported_seq":7275,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.203629+0000","last_change":"2024-09-11T05:54:01.203712+0000","last_active":"2024-09-11T05:54:01.203629+0000","last_peered":"2024-09-11T05:54:01.203629+0000","last_clean":"2024-09-11T05:54:01.203629+0000","last_became_active":"2024-09-11T05:53:10.528589+0000","last_became_peered":"2024-09-11T05:53:10.528589+0000","last_unstale":"2024-09-11T05:54:01.203629+0000","last_undegraded":"2024-09-11T05:54:01.203629+0000","last_fullsized":"2024-09-11T05:54:01.203629+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'801","last_scrub_stamp":"2024-09-11T05:54:01.203414+0000","last_deep_scrub":"171'801","last_deep_scrub_stamp":"2024-09-11T05:54:01.203414+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.203414+0000","objects_scrubbed":14,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:18:21.460069+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.001508316,"stat_sum":{"num_bytes":0,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":14,"num_read":9163,"num_read_kb":22775,"num_write":825,"num_write_kb":9319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":27222605,"num_keys_recovered":225,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":336,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:06.358206+0000","last_change":"2024-09-11T05:54:06.358260+0000","last_active":"2024-09-11T05:54:06.358206+0000","last_peered":"2024-09-11T05:54:06.358206+0000","last_clean":"2024-09-11T05:54:06.358206+0000","last_became_active":"2024-09-11T05:52:05.517814+0000","last_became_peered":"2024-09-11T05:52:05.517814+0000","last_unstale":"2024-09-11T05:54:06.358206+0000","last_undegraded":"2024-09-11T05:54:06.358206+0000","last_fullsized":"2024-09-11T05:54:06.358206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:54:06.358010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:54:06.358010+0000","last_clean_scrub_stamp":"2024-09-11T05:54:06.358010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:26.211814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011156339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"171'483","reported_seq":7422,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:03.475264+0000","last_change":"2024-09-11T05:54:03.475460+0000","last_active":"2024-09-11T05:54:03.475264+0000","last_peered":"2024-09-11T05:54:03.475264+0000","last_clean":"2024-09-11T05:54:03.475264+0000","last_became_active":"2024-09-11T05:53:43.560669+0000","last_became_peered":"2024-09-11T05:53:43.560669+0000","last_unstale":"2024-09-11T05:54:03.475264+0000","last_undegraded":"2024-09-11T05:54:03.475264+0000","last_fullsized":"2024-09-11T05:54:03.475264+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'483","last_scrub_stamp":"2024-09-11T05:54:03.475046+0000","last_deep_scrub":"171'483","last_deep_scrub_stamp":"2024-09-11T05:54:03.475046+0000","last_clean_scrub_stamp":"2024-09-11T05:54:03.475046+0000","objects_scrubbed":9,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:13:25.112345+0000","scrub_duration":15,"objects_trimmed":1,"snaptrim_duration":0.0071640180000000003,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6629,"num_read_kb":19611,"num_write":566,"num_write_kb":13391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":2938886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"171'2","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:53:59.607813+0000","last_change":"2024-09-11T05:53:59.607902+0000","last_active":"2024-09-11T05:53:59.607813+0000","last_peered":"2024-09-11T05:53:59.607813+0000","last_clean":"2024-09-11T05:53:59.607813+0000","last_became_active":"2024-09-11T05:53:35.550402+0000","last_became_peered":"2024-09-11T05:53:35.550402+0000","last_unstale":"2024-09-11T05:53:59.607813+0000","last_undegraded":"2024-09-11T05:53:59.607813+0000","last_fullsized":"2024-09-11T05:53:59.607813+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'2","last_scrub_stamp":"2024-09-11T05:53:59.607623+0000","last_deep_scrub":"171'2","last_deep_scrub_stamp":"2024-09-11T05:53:59.607623+0000","last_clean_scrub_stamp":"2024-09-11T05:53:59.607623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:03:46.734546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"171'119","reported_seq":505,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:08.415990+0000","last_change":"2024-09-11T05:54:08.416066+0000","last_active":"2024-09-11T05:54:08.415990+0000","last_peered":"2024-09-11T05:54:08.415990+0000","last_clean":"2024-09-11T05:54:08.415990+0000","last_became_active":"2024-09-11T05:52:54.928405+0000","last_became_peered":"2024-09-11T05:52:54.928405+0000","last_unstale":"2024-09-11T05:54:08.415990+0000","last_undegraded":"2024-09-11T05:54:08.415990+0000","last_fullsized":"2024-09-11T05:54:08.415990+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'119","last_scrub_stamp":"2024-09-11T05:54:08.415778+0000","last_deep_scrub":"171'119","last_deep_scrub_stamp":"2024-09-11T05:54:08.415778+0000","last_clean_scrub_stamp":"2024-09-11T05:54:08.415778+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:26.585096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"171'566","reported_seq":6237,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.519151+0000","last_change":"2024-09-11T05:54:00.519225+0000","last_active":"2024-09-11T05:54:00.519151+0000","last_peered":"2024-09-11T05:54:00.519151+0000","last_clean":"2024-09-11T05:54:00.519151+0000","last_became_active":"2024-09-11T05:53:43.557919+0000","last_became_peered":"2024-09-11T05:53:43.557919+0000","last_unstale":"2024-09-11T05:54:00.519151+0000","last_undegraded":"2024-09-11T05:54:00.519151+0000","last_fullsized":"2024-09-11T05:54:00.519151+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'566","last_scrub_stamp":"2024-09-11T05:54:00.519003+0000","last_deep_scrub":"171'566","last_deep_scrub_stamp":"2024-09-11T05:54:00.519003+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.519003+0000","objects_scrubbed":7,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:34:23.561552+0000","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.011254237,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5676,"num_read_kb":16513,"num_write":597,"num_write_kb":18171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":8713022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"171'326","reported_seq":3695,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.631312+0000","last_change":"2024-09-11T05:54:01.631486+0000","last_active":"2024-09-11T05:54:01.631312+0000","last_peered":"2024-09-11T05:54:01.631312+0000","last_clean":"2024-09-11T05:54:01.631312+0000","last_became_active":"2024-09-11T05:51:06.472099+0000","last_became_peered":"2024-09-11T05:51:06.472099+0000","last_unstale":"2024-09-11T05:54:01.631312+0000","last_undegraded":"2024-09-11T05:54:01.631312+0000","last_fullsized":"2024-09-11T05:54:01.631312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'326","last_scrub_stamp":"2024-09-11T05:54:01.631040+0000","last_deep_scrub":"171'326","last_deep_scrub_stamp":"2024-09-11T05:54:01.631040+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.631040+0000","objects_scrubbed":9,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:58:13.541305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0051173130000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":2951,"num_read_kb":13236,"num_write":255,"num_write_kb":4137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"171'774","reported_seq":6551,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.493343+0000","last_change":"2024-09-11T05:54:02.493443+0000","last_active":"2024-09-11T05:54:02.493343+0000","last_peered":"2024-09-11T05:54:02.493343+0000","last_clean":"2024-09-11T05:54:02.493343+0000","last_became_active":"2024-09-11T05:53:49.569954+0000","last_became_peered":"2024-09-11T05:53:49.569954+0000","last_unstale":"2024-09-11T05:54:02.493343+0000","last_undegraded":"2024-09-11T05:54:02.493343+0000","last_fullsized":"2024-09-11T05:54:02.493343+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'774","last_scrub_stamp":"2024-09-11T05:54:02.493147+0000","last_deep_scrub":"171'774","last_deep_scrub_stamp":"2024-09-11T05:54:02.493147+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.493147+0000","objects_scrubbed":12,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:31:36.304743+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.011383725000000001,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":11,"num_read":9010,"num_read_kb":16755,"num_write":815,"num_write_kb":11405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"171'546","reported_seq":8029,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:04.484419+0000","last_change":"2024-09-11T05:54:04.484531+0000","last_active":"2024-09-11T05:54:04.484419+0000","last_peered":"2024-09-11T05:54:04.484419+0000","last_clean":"2024-09-11T05:54:04.484419+0000","last_became_active":"2024-09-11T05:53:43.563632+0000","last_became_peered":"2024-09-11T05:53:43.563632+0000","last_unstale":"2024-09-11T05:54:04.484419+0000","last_undegraded":"2024-09-11T05:54:04.484419+0000","last_fullsized":"2024-09-11T05:54:04.484419+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'546","last_scrub_stamp":"2024-09-11T05:54:04.484200+0000","last_deep_scrub":"171'546","last_deep_scrub_stamp":"2024-09-11T05:54:04.484200+0000","last_clean_scrub_stamp":"2024-09-11T05:54:04.484200+0000","objects_scrubbed":13,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:36:42.240372+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0022360259999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8224,"num_read_kb":25972,"num_write":525,"num_write_kb":12487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":7663486,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"171'453","reported_seq":9466,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:05.612408+0000","last_change":"2024-09-11T05:54:05.612485+0000","last_active":"2024-09-11T05:54:05.612408+0000","last_peered":"2024-09-11T05:54:05.612408+0000","last_clean":"2024-09-11T05:54:05.612408+0000","last_became_active":"2024-09-11T05:53:41.551990+0000","last_became_peered":"2024-09-11T05:53:41.551990+0000","last_unstale":"2024-09-11T05:54:05.612408+0000","last_undegraded":"2024-09-11T05:54:05.612408+0000","last_fullsized":"2024-09-11T05:54:05.612408+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'453","last_scrub_stamp":"2024-09-11T05:54:05.612198+0000","last_deep_scrub":"171'453","last_deep_scrub_stamp":"2024-09-11T05:54:05.612198+0000","last_clean_scrub_stamp":"2024-09-11T05:54:05.612198+0000","objects_scrubbed":8,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:04:41.854573+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0011039960000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":8804,"num_read_kb":15047,"num_write":517,"num_write_kb":9483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":1032793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"171'571","reported_seq":5970,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:09.440991+0000","last_change":"2024-09-11T05:54:09.441069+0000","last_active":"2024-09-11T05:54:09.440991+0000","last_peered":"2024-09-11T05:54:09.440991+0000","last_clean":"2024-09-11T05:54:09.440991+0000","last_became_active":"2024-09-11T05:53:40.555028+0000","last_became_peered":"2024-09-11T05:53:40.555028+0000","last_unstale":"2024-09-11T05:54:09.440991+0000","last_undegraded":"2024-09-11T05:54:09.440991+0000","last_fullsized":"2024-09-11T05:54:09.440991+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'571","last_scrub_stamp":"2024-09-11T05:54:09.440804+0000","last_deep_scrub":"171'571","last_deep_scrub_stamp":"2024-09-11T05:54:09.440804+0000","last_clean_scrub_stamp":"2024-09-11T05:54:09.440804+0000","objects_scrubbed":8,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:21:07.066709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0040558529999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":5538,"num_read_kb":23091,"num_write":679,"num_write_kb":8926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":3095885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"171'1037","reported_seq":12893,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:03.375698+0000","last_change":"2024-09-11T05:54:03.375790+0000","last_active":"2024-09-11T05:54:03.375698+0000","last_peered":"2024-09-11T05:54:03.375698+0000","last_clean":"2024-09-11T05:54:03.375698+0000","last_became_active":"2024-09-11T05:51:05.522070+0000","last_became_peered":"2024-09-11T05:51:05.522070+0000","last_unstale":"2024-09-11T05:54:03.375698+0000","last_undegraded":"2024-09-11T05:54:03.375698+0000","last_fullsized":"2024-09-11T05:54:03.375698+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'1037","last_scrub_stamp":"2024-09-11T05:54:03.375510+0000","last_deep_scrub":"171'1037","last_deep_scrub_stamp":"2024-09-11T05:54:03.375510+0000","last_clean_scrub_stamp":"2024-09-11T05:54:03.375510+0000","objects_scrubbed":10,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:48:04.551011+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0034865120000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14883,"num_read_kb":36737,"num_write":1270,"num_write_kb":18465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"171'949","reported_seq":3896,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:05.357852+0000","last_change":"2024-09-11T05:54:05.357938+0000","last_active":"2024-09-11T05:54:05.357852+0000","last_peered":"2024-09-11T05:54:05.357852+0000","last_clean":"2024-09-11T05:54:05.357852+0000","last_became_active":"2024-09-11T05:51:05.523709+0000","last_became_peered":"2024-09-11T05:51:05.523709+0000","last_unstale":"2024-09-11T05:54:05.357852+0000","last_undegraded":"2024-09-11T05:54:05.357852+0000","last_fullsized":"2024-09-11T05:54:05.357852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'949","last_scrub_stamp":"2024-09-11T05:54:05.357650+0000","last_deep_scrub":"171'949","last_deep_scrub_stamp":"2024-09-11T05:54:05.357650+0000","last_clean_scrub_stamp":"2024-09-11T05:54:05.357650+0000","objects_scrubbed":9,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:26:40.302471+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0060758979999999997,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":6046,"num_read_kb":16311,"num_write":1495,"num_write_kb":9365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"171'1181","reported_seq":4183,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:01.509574+0000","last_change":"2024-09-11T05:54:01.509661+0000","last_active":"2024-09-11T05:54:01.509574+0000","last_peered":"2024-09-11T05:54:01.509574+0000","last_clean":"2024-09-11T05:54:01.509574+0000","last_became_active":"2024-09-11T05:52:54.925604+0000","last_became_peered":"2024-09-11T05:52:54.925604+0000","last_unstale":"2024-09-11T05:54:01.509574+0000","last_undegraded":"2024-09-11T05:54:01.509574+0000","last_fullsized":"2024-09-11T05:54:01.509574+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'1181","last_scrub_stamp":"2024-09-11T05:54:01.509373+0000","last_deep_scrub":"171'1181","last_deep_scrub_stamp":"2024-09-11T05:54:01.509373+0000","last_clean_scrub_stamp":"2024-09-11T05:54:01.509373+0000","objects_scrubbed":10,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:31:38.010182+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001435303,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6847,"num_read_kb":23341,"num_write":1773,"num_write_kb":11275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15596104,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"171'599","reported_seq":5056,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:04.324873+0000","last_change":"2024-09-11T05:54:04.325056+0000","last_active":"2024-09-11T05:54:04.324873+0000","last_peered":"2024-09-11T05:54:04.324873+0000","last_clean":"2024-09-11T05:54:04.324873+0000","last_became_active":"2024-09-11T05:51:05.522211+0000","last_became_peered":"2024-09-11T05:51:05.522211+0000","last_unstale":"2024-09-11T05:54:04.324873+0000","last_undegraded":"2024-09-11T05:54:04.324873+0000","last_fullsized":"2024-09-11T05:54:04.324873+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'599","last_scrub_stamp":"2024-09-11T05:54:04.324638+0000","last_deep_scrub":"171'599","last_deep_scrub_stamp":"2024-09-11T05:54:04.324638+0000","last_clean_scrub_stamp":"2024-09-11T05:54:04.324638+0000","objects_scrubbed":7,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:24:45.205574+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001757485,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":5699,"num_read_kb":12216,"num_write":712,"num_write_kb":5629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"171'746","reported_seq":4052,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:00.190007+0000","last_change":"2024-09-11T05:54:00.190039+0000","last_active":"2024-09-11T05:54:00.190007+0000","last_peered":"2024-09-11T05:54:00.190007+0000","last_clean":"2024-09-11T05:54:00.190007+0000","last_became_active":"2024-09-11T05:52:55.539380+0000","last_became_peered":"2024-09-11T05:52:55.539380+0000","last_unstale":"2024-09-11T05:54:00.190007+0000","last_undegraded":"2024-09-11T05:54:00.190007+0000","last_fullsized":"2024-09-11T05:54:00.190007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'746","last_scrub_stamp":"2024-09-11T05:54:00.189931+0000","last_deep_scrub":"171'746","last_deep_scrub_stamp":"2024-09-11T05:54:00.189931+0000","last_clean_scrub_stamp":"2024-09-11T05:54:00.189931+0000","objects_scrubbed":5,"log_size":746,"log_dups_size":0,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:41:29.094590+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0020568969999999998,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5351,"num_read_kb":14612,"num_write":938,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":6817358,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"171'481","reported_seq":14400,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-11T05:54:02.401690+0000","last_change":"2024-09-11T05:54:02.401768+0000","last_active":"2024-09-11T05:54:02.401690+0000","last_peered":"2024-09-11T05:54:02.401690+0000","last_clean":"2024-09-11T05:54:02.401690+0000","last_became_active":"2024-09-11T05:51:05.521843+0000","last_became_peered":"2024-09-11T05:51:05.521843+0000","last_unstale":"2024-09-11T05:54:02.401690+0000","last_undegraded":"2024-09-11T05:54:02.401690+0000","last_fullsized":"2024-09-11T05:54:02.401690+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'481","last_scrub_stamp":"2024-09-11T05:54:02.401505+0000","last_deep_scrub":"171'481","last_deep_scrub_stamp":"2024-09-11T05:54:02.401505+0000","last_clean_scrub_stamp":"2024-09-11T05:54:02.401505+0000","objects_scrubbed":10,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:20:21.933542+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0029824199999999999,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":14130,"num_read_kb":26046,"num_write":513,"num_write_kb":15066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":275,"num_read":220223,"num_read_kb":578369,"num_write":24059,"num_write_kb":303275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4773,"num_bytes_recovered":140111398,"num_keys_recovered":613,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20398,"ondisk_log_size":20398,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542191,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78188,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94293652,"statfs":{"total":96636764160,"available":96556699648,"internally_reserved":0,"allocated":2650112,"data_stored":1573353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542192,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51240,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320600,"statfs":{"total":96636764160,"available":96584294400,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33196,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338644,"statfs":{"total":96636764160,"available":96602771456,"internally_reserved":0,"allocated":2060288,"data_stored":983004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69292,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302548,"statfs":{"total":96636764160,"available":96565809152,"internally_reserved":0,"allocated":2060288,"data_stored":983004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542191,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59560,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312280,"statfs":{"total":96636764160,"available":96575774720,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542190,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41512,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330328,"statfs":{"total":96636764160,"available":96594255872,"internally_reserved":0,"allocated":2056192,"data_stored":982985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542190,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2064384,"data_stored":983023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542192,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60140,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311700,"statfs":{"total":96636764160,"available":96575180800,"internally_reserved":0,"allocated":2650112,"data_stored":1573353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:54:21.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T05:54:21.448 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T05:54:21.448 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T05:54:21.448 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T05:54:21.448 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:21.536 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T05:54:21.536 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T05:54:21.536 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:21.613 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T05:54:21.614 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T05:54:21.614 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:21.705 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T05:54:21.706 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T05:54:21.706 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:27.808 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T05:54:27.808 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T05:54:27.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:27.886 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T05:54:27.887 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T05:54:27.887 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:27.968 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T05:54:27.968 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-11T05:54:27.968 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:28.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-11T05:54:34.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-11T05:54:34.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-11T05:54:34.071 INFO:tasks.ceph.osd.6:Stopped 2024-09-11T05:54:34.071 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-11T05:54:34.071 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.141 INFO:tasks.ceph.osd.7:Stopped 2024-09-11T05:54:34.142 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T05:54:34.142 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-11T05:54:34.142 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.186 INFO:tasks.ceph.mgr.y:Stopped 2024-09-11T05:54:34.186 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T05:54:34.186 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.217 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T05:54:34.217 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T05:54:34.217 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:54:34.217 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.235 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:54:34.235 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T05:54:34.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.295 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T05:54:34.295 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:54:34.296 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:54:34.314 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:54:34.314 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T05:54:34.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-11T05:54:34.353 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:54:34.354 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T05:54:34.725 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:54:34.725 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T05:54:34.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:54:34.941 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T05:54:35.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:54:35.109 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T05:54:35.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:54:35.283 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T05:54:35.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:54:35.574 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T05:54:35.790 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:54:35.791 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-11T05:54:35.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:54:35.983 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-11T05:54:36.186 INFO:tasks.ceph:Archiving mon data... 2024-09-11T05:54:36.187 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/data/mon.a.tgz 2024-09-11T05:54:36.187 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-11T05:54:36.193 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.1WWEutHu7W 2024-09-11T05:54:36.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.1WWEutHu7W 2024-09-11T05:54:36.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.1WWEutHu7W 2024-09-11T05:54:36.401 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.1WWEutHu7W is 397KB 2024-09-11T05:54:36.463 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.1WWEutHu7W 2024-09-11T05:54:36.468 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/data/mon.c.tgz 2024-09-11T05:54:36.468 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-11T05:54:36.517 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.ZycC4jb1aa 2024-09-11T05:54:36.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ZycC4jb1aa 2024-09-11T05:54:36.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.ZycC4jb1aa 2024-09-11T05:54:36.760 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.ZycC4jb1aa is 536KB 2024-09-11T05:54:36.823 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.ZycC4jb1aa 2024-09-11T05:54:36.828 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/data/mon.b.tgz 2024-09-11T05:54:36.828 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-09-11T05:54:36.834 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.Z1HoskgB28 2024-09-11T05:54:36.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Z1HoskgB28 2024-09-11T05:54:37.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.Z1HoskgB28 2024-09-11T05:54:37.074 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.Z1HoskgB28 is 536KB 2024-09-11T05:54:37.136 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.Z1HoskgB28 2024-09-11T05:54:37.141 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T05:54:37.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T05:54:37.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T05:54:37.203 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T05:54:37.246 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T05:54:37.246 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T05:54:37.248 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi165/crash 2024-09-11T05:54:37.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T05:54:37.262 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi172/crash 2024-09-11T05:54:37.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T05:54:37.302 INFO:tasks.ceph:Compressing logs... 2024-09-11T05:54:37.302 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T05:54:37.310 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T05:54:37.334 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36606.log 2024-09-11T05:54:37.334 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-09-11T05:54:37.335 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32579.log 2024-09-11T05:54:37.336 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36606.log: /var/log/ceph/ceph-client.admin.38764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36606.log.gz 2024-09-11T05:54:37.336 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-09-11T05:54:37.336 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32579.log.gz 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32369.log 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39071.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz -5 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34091.log 2024-09-11T05:54:37.337 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32369.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34241.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34091.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32356.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34241.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35943.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37213.log 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32356.log.gz 2024-09-11T05:54:37.340 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35943.log.gz 2024-09-11T05:54:37.341 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37300.log 2024-09-11T05:54:37.343 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35193.log 2024-09-11T05:54:37.343 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37213.log.gz 2024-09-11T05:54:37.343 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2024-09-11T05:54:37.343 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37300.log.gz 2024-09-11T05:54:37.343 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37603.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35193.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33187.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37603.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:gzip/var/log/ceph/ceph-client.admin.38957.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40391.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.0.33187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36549.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40391.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40391.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.0.33187.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36549.log.gz 2024-09-11T05:54:37.344 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33535.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37942.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.33535.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21493.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33535.log.gz 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36354.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37942.log.gz 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph.tmp-client.admin.21493.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21493.log.gz 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36422.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36354.log.gz 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36422.log.gz 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36747.log 2024-09-11T05:54:37.350 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36747.log.gz 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40313.log 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41535.log 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-09-11T05:54:37.351 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40313.log.gz 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39200.log 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36229.log 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41535.log.gz 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39200.log.gz 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36229.log.gz 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-09-11T05:54:37.352 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37426.log 2024-09-11T05:54:37.365 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34540.log 2024-09-11T05:54:37.365 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-09-11T05:54:37.365 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37426.log.gz 2024-09-11T05:54:37.365 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36271.log 2024-09-11T05:54:37.365 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34540.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36271.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35185.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31657.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35185.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33098.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.31657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35575.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31657.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33098.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33197.log 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35575.log.gz 2024-09-11T05:54:37.366 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32360.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36150.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33197.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32360.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32221.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36150.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33046.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32221.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37238.log 2024-09-11T05:54:37.367 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33046.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37238.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37072.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34356.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37072.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37046.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34356.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37046.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34420.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-09-11T05:54:37.368 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34420.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41379.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32358.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41379.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32803.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32358.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35367.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32803.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35367.log.gz 2024-09-11T05:54:37.369 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35876.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36011.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35876.log.gz 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33949.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37010.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36011.log.gz 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33949.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33949.log.gz --verbose 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr: -- /var/log/ceph/ceph-client.admin.37577.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37010.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37010.log.gz /var/log/ceph/ceph-client.admin.34247.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37339.log 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34247.log.gz 2024-09-11T05:54:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32560.log 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26423.log 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26399.log 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26423.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26423.log.gz 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-11T05:54:37.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26399.log.gz 2024-09-11T05:54:37.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T05:54:37.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26351.log 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26447.log 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.26351.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26351.log.gz 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26375.log 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26447.log.gz 2024-09-11T05:54:37.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26471.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26375.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26311.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26471.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26471.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26311.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26495.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.26495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26495.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T05:54:37.390 INFO:teuthology.orchestra.run.smithi172.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37339.log.gz 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32560.log.gz 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34357.log 2024-09-11T05:54:37.402 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34357.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.34357.log.gz /var/log/ceph/ceph-client.admin.39492.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40732.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40732.log.gz 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34973.log 2024-09-11T05:54:37.403 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35192.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35602.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34973.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36656.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35192.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35602.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36656.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33020.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32371.log 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-09-11T05:54:37.404 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33020.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34036.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32371.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34036.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26500.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36797.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39413.log 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26500.log.gz 2024-09-11T05:54:37.405 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36797.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41353.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39413.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41587.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32552.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41353.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41587.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33692.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32552.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-09-11T05:54:37.406 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34193.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33692.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32968.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34193.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32968.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34471.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-09-11T05:54:37.407 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34471.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33423.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33124.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33423.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35773.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33124.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35773.log.gz 2024-09-11T05:54:37.408 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40911.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40911.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35135.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39442.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35135.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35135.log.gz 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2024-09-11T05:54:37.409 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32768.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36474.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41767.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32768.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36474.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36015.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40365.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41767.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36015.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40365.log: /var/log/ceph/ceph-client.admin.40723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40365.log.gz 2024-09-11T05:54:37.410 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39968.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40514.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39968.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35029.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40514.log: /var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36186.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40514.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35029.log.gz 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38332.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36102.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36186.log: gzip/var/log/ceph/ceph-client.admin.38332.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32370.log 2024-09-11T05:54:37.411 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38332.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36186.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36102.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32569.log 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32370.log: /var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32370.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32569.log.gz 2024-09-11T05:54:37.412 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39736.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38187.log: /var/log/ceph/ceph-client.admin.39736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33763.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39736.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41665.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33990.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33763.log: /var/log/ceph/ceph-client.admin.41665.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33763.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41665.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33990.log.gz 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39682.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-09-11T05:54:37.413 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.1.log: gzip/var/log/ceph/ceph-client.admin.39682.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39682.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32767.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32767.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33617.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33617.log.gz 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36448.log 2024-09-11T05:54:37.414 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37395.log 2024-09-11T05:54:37.433 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39558.log: /var/log/ceph/ceph-client.admin.36448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-09-11T05:54:37.433 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36448.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39649.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34134.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37395.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39649.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34134.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37785.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32372.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40027.log: /var/log/ceph/ceph-client.admin.37785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37785.log.gz 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34821.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-09-11T05:54:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32372.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34821.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38435.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32266.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38435.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39341.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38291.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32266.log: /var/log/ceph/ceph-client.admin.39341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32266.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39341.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37136.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41743.log 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38291.log: /var/log/ceph/ceph-client.admin.37136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38291.log.gz 2024-09-11T05:54:37.435 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37136.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38473.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41743.log: /var/log/ceph/ceph-client.admin.40518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41743.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32565.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35826.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38473.log: /var/log/ceph/ceph-client.admin.32565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38473.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32565.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33476.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35826.log.gz 2024-09-11T05:54:37.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33476.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34732.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34160.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34732.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33874.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34160.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32994.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33874.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35326.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32994.log: /var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33154.log 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32994.log.gz 2024-09-11T05:54:37.437 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35326.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39108.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33154.log: /var/log/ceph/ceph-client.admin.33072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33718.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33154.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39108.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34424.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33718.log: /var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37494.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33718.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34424.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.33825.log 2024-09-11T05:54:37.438 INFO:teuthology.orchestra.run.smithi165.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34424.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41717.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37494.log: /var/log/ceph/ceph-client.admin.33825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38497.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37494.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33825.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34736.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41717.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38497.log: /var/log/ceph/ceph-client.admin.34736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38497.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34736.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40443.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36864.log 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-09-11T05:54:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36864.log.gz 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37993.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37993.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33366.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34638.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33615.log 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33366.log: /var/log/ceph/ceph-client.admin.34638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33366.log.gz 2024-09-11T05:54:37.440 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34638.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32292.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33615.log: /var/log/ceph/ceph-client.admin.35094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33615.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32781.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32190.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32292.log: /var/log/ceph/ceph-client.admin.32781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32292.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32781.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32190.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-09-11T05:54:37.441 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32246.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41561.log 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32246.log.gz 2024-09-11T05:54:37.451 INFO:teuthology.orchestra.run.smithi165.stderr:gzip 93.9% -5 --verbose -- replaced with /var/log/ceph/ceph.audit.log.gz -- 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr: /var/log/ceph/ceph-client.admin.32760.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41561.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36349.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32760.log: /var/log/ceph/ceph-client.admin.36349.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32760.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36349.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32367.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36917.log 2024-09-11T05:54:37.452 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38382.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32367.log.gz 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36917.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36979.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38382.log.gz 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37759.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32572.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36979.log.gz 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37759.log.gz 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34613.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32572.log.gz 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-09-11T05:54:37.453 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34613.log: /var/log/ceph/ceph-client.admin.40183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34613.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26347.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26347.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36135.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32784.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36135.log.gz 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41275.log 2024-09-11T05:54:37.454 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32784.log.gz 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35947.log 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41275.log.gz 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35947.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38154.log 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.460 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35393.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38154.log.gz 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40287.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35393.log.gz 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37900.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40287.log: /var/log/ceph/ceph-client.admin.37664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40287.log.gz 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32783.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37502.log 2024-09-11T05:54:37.461 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37900.log: /var/log/ceph/ceph-client.admin.32783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37900.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32783.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41327.log 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37502.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41327.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38593.log 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-09-11T05:54:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33325.log 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38593.log.gz 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33510.log 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38245.log 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33325.log.gz 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.33510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33510.log.gz 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38245.log.gz 2024-09-11T05:54:37.466 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36312.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36681.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36312.log.gz 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36681.log.gz 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38863.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2024-09-11T05:54:37.467 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38863.log.gz 2024-09-11T05:54:37.469 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-09-11T05:54:37.469 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41431.log 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.37789.log: /var/log/ceph/ceph-client.admin.39943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32580.log 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41431.log.gz 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32580.log.gz 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.41639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2024-09-11T05:54:37.470 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32318.log 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34315.log 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.40235.log: /var/log/ceph/ceph-client.admin.32318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32318.log.gz 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36792.log 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34920.log 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34315.log.gz 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.36792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36792.log.gz 2024-09-11T05:54:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35002.log 2024-09-11T05:54:37.473 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26600.log 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.34920.log: /var/log/ceph/ceph-client.admin.35002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34920.log.gz 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35002.log.gz 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34501.log 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39004.log 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.26600.log: /var/log/ceph/ceph-client.admin.34501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26600.log.gz 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34501.log.gz 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35880.log 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.39004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39004.log.gz 2024-09-11T05:54:37.475 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.35880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35880.log.gz 2024-09-11T05:54:37.476 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.38679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-09-11T05:54:37.495 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T05:54:37.644 INFO:teuthology.orchestra.run.smithi165.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T05:54:37.644 INFO:teuthology.orchestra.run.smithi165.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T05:54:38.060 INFO:teuthology.orchestra.run.smithi172.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-11T05:54:38.251 INFO:teuthology.orchestra.run.smithi165.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T05:54:39.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-09-11T05:54:39.091 INFO:teuthology.orchestra.run.smithi165.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T05:54:39.647 INFO:teuthology.orchestra.run.smithi172.stderr: 95.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T05:54:40.503 INFO:teuthology.orchestra.run.smithi165.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T05:54:41.041 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T05:54:41.882 INFO:teuthology.orchestra.run.smithi165.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T05:54:41.884 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-11T05:54:41.884 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m4.572s 2024-09-11T05:54:41.884 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m10.612s 2024-09-11T05:54:41.884 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.499s 2024-09-11T05:54:42.158 INFO:teuthology.orchestra.run.smithi172.stderr: 95.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-11T05:54:42.159 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-11T05:54:42.159 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m4.803s 2024-09-11T05:54:42.159 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m11.043s 2024-09-11T05:54:42.159 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.498s 2024-09-11T05:54:42.160 INFO:tasks.ceph:Archiving logs... 2024-09-11T05:54:42.160 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi165/log 2024-09-11T05:54:42.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T05:54:43.503 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi172/log 2024-09-11T05:54:43.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T05:54:44.636 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T05:54:44.648 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T05:54:44.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T05:54:44.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T05:54:44.720 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T05:54:44.731 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T05:54:44.731 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:54:44.734 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout:*hv01.front.sepi 67.205.162.81 3 u 64 64 77 0.149 -0.107 1.198 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout:+hv02.front.sepi 96.245.170.99 3 u 63 64 77 0.089 +0.297 0.843 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 77 0.087 -1.350 0.855 2024-09-11T05:54:44.746 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:54:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:54:44.747 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-09-11T05:54:44.747 INFO:teuthology.orchestra.run.smithi172.stdout:+hv01.front.sepi 67.205.162.81 3 u - 64 177 0.073 -1.024 1.120 2024-09-11T05:54:44.748 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 96.245.170.99 3 u 61 64 77 0.093 -1.945 1.928 2024-09-11T05:54:44.748 INFO:teuthology.orchestra.run.smithi172.stdout:*hv03.front.sepi 74.6.168.72 3 u 2 64 177 0.094 -1.948 1.147 2024-09-11T05:54:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:54:44.748 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T05:54:44.759 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T05:54:44.760 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T05:54:44.775 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T05:54:44.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T05:54:44.804 INFO:teuthology.task.internal:Duration was 849.767871 seconds 2024-09-11T05:54:44.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T05:54:44.819 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T05:54:44.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T05:54:44.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T05:54:44.860 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T05:54:44.860 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-09-11T05:54:44.860 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T05:54:44.904 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-09-11T05:54:44.904 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T05:54:44.915 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T05:54:44.915 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T05:54:44.953 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T05:54:44.991 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T05:54:44.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T05:54:45.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T05:54:45.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T05:54:45.111 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T05:54:45.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T05:54:45.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T05:54:45.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T05:54:45.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:54:45.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:54:45.173 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-09-11T05:54:45.182 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-09-11T05:54:45.278 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:54:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:54:45.283 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:54:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:54:45.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T05:54:45.334 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T05:54:45.335 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi165 2024-09-11T05:54:45.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T05:54:45.361 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900581/remote/smithi172 2024-09-11T05:54:45.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T05:54:45.389 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T05:54:45.389 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T05:54:45.402 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T05:54:45.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T05:54:45.447 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T05:54:45.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T05:54:45.462 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T05:54:45.462 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T05:54:45.465 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T05:54:45.468 INFO:teuthology.orchestra.run.smithi165.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 05:54 /home/ubuntu/cephtest 2024-09-11T05:54:45.482 INFO:teuthology.orchestra.run.smithi172.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 05:54 /home/ubuntu/cephtest 2024-09-11T05:54:45.483 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T05:54:45.495 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T05:54:45.528 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writethrough} duration: 849.7678709030151 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-11T05:54:45.528 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T05:54:45.620 INFO:teuthology.run:pass